经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置：技术经验 » 数据库/运维 » MongoDB » 查看文章
【最佳实践】MongoDB导出导入数据
来源：cnblogs　　作者：likingzi　　时间：2023/10/9 9:22:11　　对本文有异议

首先说一下这个3节点MongoDB集群各个维度的数据规模：
1、dataSize: 1.9T
2、storageSize: 600G
3、全量备份-加压缩开关：186G，耗时 8h
4、全量备份-不加压缩开关：1.8T，耗时 4h27m
具体导出的语法比较简单，此处不再赘述，本文重点描述导入的优化过程，最后给出导入的最佳实践。

■ 2023-09-13T20:00 第1次4并发导入测试

  1. mongorestore --port=20000 -uadmin -p'passwd' --authenticationDatabase=admin --numInsertionWorkersPerCollection=4 --bypassDocumentValidation -d likingtest /u01/nfs/xxxxx_mongodb/10.1.1.1/20230913/likingtest >> 10.2.2.2.log 2>&1 &
  2. tail -100f /u01/nfs/xxxxx_mongodb/10.1.1.1/20230913/10.2.2.2.log
  3. 以上导入：
  4. 2023-09-13T21:59:55.452+0800    The --db and --collection flags are deprecated for this use-case; please use --nsInclude instead, i.e. with --nsInclude=${DATABASE}.${COLLECTION}
  5. 2023-09-13T21:59:55.452+0800    building a list of collections to restore from /u01/nfs/xxxxx_mongodb/10.1.1.1/20230913/likingtest dir
  6. 2023-09-13T21:59:55.466+0800    reading metadata for likingtest.oprceConfiguration from /u01/nfs/xxxxx_mongodb/10.1.1.1/20230913/likingtest/oprceConfiguration.metadata.json
  7. 2023-09-13T21:59:55.478+0800    reading metadata for likingtest.oprceDataObj from /u01/nfs/xxxxx_mongodb/10.1.1.1/20230913/likingtest/oprceDataObj.metadata.json
  8. 2023-09-13T21:59:55.491+0800    reading metadata for likingtest.oprcesDataObjInit from /u01/nfs/xxxxx_mongodb/10.1.1.1/20230913/likingtest/oprcesDataObjInit.metadata.json
  9. 2023-09-13T21:59:55.503+0800    reading metadata for likingtest.role from /u01/nfs/xxxxx_mongodb/10.1.1.1/20230913/likingtest/role.metadata.json
  10. 2023-09-13T21:59:55.508+0800    reading metadata for likingtest.activityConfiguration from /u01/nfs/xxxxx_mongodb/10.1.1.1/20230913/likingtest/activityConfiguration.metadata.json
  11. 2023-09-13T21:59:55.511+0800    reading metadata for likingtest.history_task from /u01/nfs/xxxxx_mongodb/10.1.1.1/20230913/likingtest/history_task.metadata.json
  12. 2023-09-13T21:59:55.512+0800    reading metadata for likingtest.resOutRelDataSnapshot from /u01/nfs/xxxxx_mongodb/10.1.1.1/20230913/likingtest/resOutRelDataSnapshot.metadata.json
  13. 2023-09-13T21:59:55.520+0800    reading metadata for likingtest.snapshotResource from /u01/nfs/xxxxx_mongodb/10.1.1.1/20230913/likingtest/snapshotResource.metadata.json
  14. 2023-09-13T21:59:55.524+0800    reading metadata for likingtest.oprceDataObjDraft from /u01/nfs/xxxxx_mongodb/10.1.1.1/20230913/likingtest/oprceDataObjDraft.metadata.json
  15. 2023-09-13T21:59:55.526+0800    reading metadata for likingtest.oprceDataObjInit from /u01/nfs/xxxxx_mongodb/10.1.1.1/20230913/likingtest/oprceDataObjInit.metadata.json
  16. 2023-09-13T21:59:55.761+0800    restoring likingtest.snapshotResource from /u01/nfs/xxxxx_mongodb/10.1.1.1/20230913/likingtest/snapshotResource.bson
  17. ...
  18. 2023-09-13T22:00:01.451+0800    [........................]      likingtest.oprceDataObj   408MB/1205GB    (0.0%)
  19. ...
  20. 2023-09-13T21:59:58.323+0800    finished restoring likingtest.oprceDataObjDraft (1559 documents, 0 failures)
  21. 2023-09-13T22:00:01.034+0800    finished restoring likingtest.resOutRelDataSnapshot (34426 documents, 0 failures)
  22. 2023-09-13T22:00:01.559+0800    finished restoring likingtest.history_task (3629 documents, 0 failures)
  23. 2023-09-13T22:00:02.086+0800    finished restoring likingtest.activityConfiguration (974 documents, 0 failures)
  24. 2023-09-13T22:00:02.293+0800    finished restoring likingtest.oprceConfiguration (162 documents, 0 failures)
  25. 2023-09-13T22:00:02.529+0800    finished restoring likingtest.oprcesDataObjInit (4 documents, 0 failures)
  26. 2023-09-13T22:00:02.857+0800    finished restoring likingtest.role (10 documents, 0 failures)
  27. 2023-09-13T22:00:29.153+0800    [########################]  likingtest.snapshotResource  2.04GB/2.04GB  (100.0%)
  28. 2023-09-13T22:00:29.155+0800    finished restoring likingtest.snapshotResource (50320 documents, 0 failures)
  29. ...
  30. 2023-09-14T00:18:58.451+0800    [############............]      likingtest.oprceDataObj  651GB/1205GB   (54.0%)
  31. 2023-09-14T00:18:59.857+0800    [########################]  likingtest.oprceDataObjInit  635GB/635GB  (100.0%)
  32. 2023-09-14T00:18:59.888+0800    finished restoring likingtest.oprceDataObjInit (43776648 documents, 0 failures)
  33. ...
  34. 2023-09-14T02:05:58.904+0800    [########################]      likingtest.oprceDataObj  1205GB/1205GB  (100.0%)
  35. 2023-09-14T02:05:58.937+0800    finished restoring likingtest.oprceDataObj (53311330 documents, 0 failures)
  36. 2023-09-14T02:05:58.945+0800    no indexes to restore for collection likingtest.activityConfiguration
  37. 2023-09-14T02:05:58.945+0800    no indexes to restore for collection likingtest.history_task
  38. 2023-09-14T02:05:58.945+0800    restoring indexes for collection likingtest.oprcesDataObjInit from metadata
  39. 2023-09-14T02:05:58.976+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowId_1_activityConfiguration.activityNameEn_1", "ns":"likingtest.oprcesDataObjInit", "v":2}, Key:primitive.D{primitive.E{Key:"flowId", Value:1}, primitive.E{Key:"activityConfiguration.activityNameEn", Value:1}}, PartialFilterExpression:primitive.D(nil)}
  40. 2023-09-14T02:05:58.976+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"oprceInfo.oprceInstID_1_activityInfo.activityInstID_1_workitemInfo.workItemID_1", "ns":"likingtest.oprcesDataObjInit", "v":2}, Key:primitive.D{primitive.E{Key:"oprceInfo.oprceInstID", Value:1}, primitive.E{Key:"activityInfo.activityInstID", Value:1}, primitive.E{Key:"workitemInfo.workItemID", Value:1}}, PartialFilterExpression:primitive.D(nil)}
  41. 2023-09-14T02:05:58.976+0800    no indexes to restore for collection likingtest.role
  42. 2023-09-14T02:05:58.976+0800    no indexes to restore for collection likingtest.snapshotResource
  43. 2023-09-14T02:05:58.976+0800    no indexes to restore for collection likingtest.oprceDataObjDraft
  44. 2023-09-14T02:05:58.976+0800    restoring indexes for collection likingtest.oprceDataObjInit from metadata
  45. 2023-09-14T02:05:58.976+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"oprceInfo.oprceInstID_1_activityInfo.activityInstID_1_workitemInfo.workItemID_1", "ns":"likingtest.oprceDataObjInit", "v":2}, Key:primitive.D{primitive.E{Key:"oprceInfo.oprceInstID", Value:1}, primitive.E{Key:"activityInfo.activityInstID", Value:1}, primitive.E{Key:"workitemInfo.workItemID", Value:1}}, PartialFilterExpression:primitive.D(nil)}
  46. 2023-09-14T02:05:58.976+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowNo_1", "ns":"likingtest.oprceDataObjInit", "v":2}, Key:primitive.D{primitive.E{Key:"flowNo", Value:1}}, PartialFilterExpression:primitive.D(nil)}
  47. 2023-09-14T02:05:58.976+0800    no indexes to restore for collection likingtest.oprceConfiguration
  48. 2023-09-14T02:05:58.976+0800    no indexes to restore for collection likingtest.resOutRelDataSnapshot
  49. 2023-09-14T02:05:58.976+0800    restoring indexes for collection likingtest.oprceDataObj from metadata
  50. 2023-09-14T02:05:58.976+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowId_1_activityConfiguration.activityNameEn_1", "ns":"likingtest.oprceDataObj", "v":2}, Key:primitive.D{primitive.E{Key:"flowId", Value:1}, primitive.E{Key:"activityConfiguration.activityNameEn",Value:1}}, PartialFilterExpression:primitive.D(nil)}
  51. 2023-09-14T02:05:58.976+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowNo_1", "ns":"likingtest.oprceDataObj", "v":2}, Key:primitive.D{primitive.E{Key:"flowNo", Value:1}}, PartialFilterExpression:primitive.D(nil)}
  52. 2023-09-14T02:05:58.976+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"oprceInfo.oprceInstID_1_activityInfo.activityInstID_1_workitemInfo.workItemID_1", "ns":"likingtest.oprceDataObj", "v":2}, Key:primitive.D{primitive.E{Key:"oprceInfo.oprceInstID", Value:1}, primitive.E{Key:"activityInfo.activityInstID", Value:1}, primitive.E{Key:"workitemInfo.workItemID", Value:1}}, PartialFilterExpression:primitive.D(nil)}
  53. 2023-09-14T02:05:58.976+0800    index: &idx.IndexDocument{Options:primitive.M{"name":"flowId_1_activityConfiguration.activityNameEn_1", "ns":"likingtest.oprceDataObjInit", "v":2}, Key:primitive.D{primitive.E{Key:"flowId", Value:1}, primitive.E{Key:"activityConfiguration.activityNameEn", Value:1}}, PartialFilterExpression:primitive.D(nil)}
  54. 2023-09-14T03:45:47.152+0800    97179062 document(s) restored successfully. 0 document(s) failed to restore.

可见：
1、配置并发参数 --numInsertionWorkersPerCollection=4 和 检查参数 bypassDocumentValidation 后，restore速度大大提升，1.2T 的一个大集合 oprceDataObj，由原来默认restore方式约 12h，降为：4h
2、restore完所有数据以后，最后再restore索引，restore索引还是需要一定的时间，本次耗时：1h40m【注：实际没有成功，索引并未生效】
3、新版本的 -d -c 参数需统一修改为：--nsInclude --nsFrom= --nsTo=

■ 2023-09-14T10:40 第2次8并发导入测试

  1. mongorestore --port=20000 -uadmin -p'passwd' --authenticationDatabase=admin --numInsertionWorkersPerCollection=8 --bypassDocumentValidation -d likingtest /u01/nfs/xxxxx_mongodb/10.1.1.1/20230914/likingtest >> 10.2.2.2.log 2>&1 &
  2. tail -100f /u01/nfs/xxxxx_mongodb/10.1.1.1/20230914/10.2.2.2.log
  3. ---
  4. 2023-09-14T10:40:45.492+0800    The --db and --collection flags are deprecated for this use-case; please use --nsInclude instead, i.e. with --nsInclude=${DATABASE}.${COLLECTION}
  5. ...
  6. 2023-09-14T10:40:48.493+0800    [........................]       likingtest.oprceDataObj   112MB/1208GB    (0.0%)
  7. ...
  8. 2023-09-14T12:57:34.859+0800    [########################]       likingtest.oprceDataObj  1208GB/1208GB  (100.0%)
  9. 2023-09-14T12:57:34.867+0800    finished restoring likingtest.oprceDataObj (53413481 documents, 0 failures)

可见：
1、配置并发参数 --numInsertionWorkersPerCollection=8 和 检查参数 --bypassDocumentValidation 后，restore速度再次大大提升，1.2T的一个大集合 oprceDataObj，由原来默认restore方式约 12h，降为：2h17m
2、本次恢复采用nfs备份恢复，一台8C的虚机，8并发恢复时cpu占用约40%，网络接收速度300MB/s左右，本地磁盘写入速度在30-200MB/s左右，可见网络带段不是瓶颈。可以预见，如果采用更高的主机配置，尤其是IO更好的磁盘，resotore时间必将更少。

■ 2023-09-14T16:10 第3次12并发导入测试

【注意】由于新版本mongorestore摒弃了-d -c参数，虽然可用但使用不够灵活，因此需使用新参数--nsInclude，对于该参数的使用，摸索了多次才找到使用的限制条件，即 directory 必须为数据库备份的根目录/上一级目录，而不是 数据库目录！即类似 dumpdir/20230914，而不是 dumpdir/20230914/database！这是一个巨大的坑，切记！当然，这个目录下一定不能有其他不可识别的文件，否则也会报错。

  1. mongorestore --port=20000 -uadmin -p'passwd' --authenticationDatabase=admin --numInsertionWorkersPerCollection=12 --bypassDocumentValidation --nsInclude="likingtest.*" /u01/nfs/xxxxx_mongodb/10.1.1.1/20230914 > 20230914.10.2.2.2-3.log 2>&1 &
  2. tail -100f /u01/nfs/xxxxx_mongodb/10.1.1.1/20230914.10.2.2.2-3.log
  3. ---
  4. 2023-09-14T16:10:19.245+0800    preparing collections to restore from
  5. ...
  6. 2023-09-14T18:18:18.996+0800    [########################]  likingtest.oprceDataObj  1208GB/1208GB  (100.0%)
  7. 2023-09-14T18:18:19.014+0800    finished restoring likingtest.oprceDataObj (53413481 documents, 0 failures)

可见：
1、并发由 8 增至 12 并无效率提升，结论是 6-8 个并发就可以，这一点与oracle的并发导入设置为 6 基本是最佳实践类似。
2、本次恢复采用nfs备份恢复，一台8C的虚机，12并发恢复时cpu占用约60%，网络接收速度300MB/s左右，本地磁盘写入速度在30-500MB/s左右，可见网络带段不是瓶颈。可以预见，如果采用更高的主机配置，尤其是IO更好的磁盘，resotore时间必将更少。
3、关于索引的restore，restore时首先恢复数据，最后再创建索引，比较大的集合的索引创建还是需要较多的时间：

  1.       currentOpTime: '2023-09-14T20:23:59.435+08:00',
  2. ...
  3.       command: {
  4.         createIndexes: 'oprceDataObj',
  5.         indexes: [
  6.           {
  7.             key: { flowId: 1, 'activityConfiguration.activityNameEn': 1 },
  8.             name: 'flowId_1_activityConfiguration.activityNameEn_1',
  9.             ns: 'likingtest.oprceDataObj'
  10.           },
  11.           {
  12.             key: { flowNo: 1 },
  13.             name: 'flowNo_1',
  14.             ns: 'likingtest.oprceDataObj'
  15.           },
  16.           {
  17.             key: {
  18.               'oprceInfo.oprceInstID': 1,
  19.               'activityInfo.activityInstID': 1,
  20.               'workitemInfo.workItemID': 1
  21.             },
  22.             name: 'oprceInfo.oprceInstID_1_activityInfo.activityInstID_1_workitemInfo.workItemID_1',
  23.             ns: 'likingtest.oprceDataObj'
  24.           }
  25.         ],
  26. .....
  27.       currentOpTime: '2023-09-14T20:23:59.489+08:00',
  28. ...
  29.       command: {
  30.         createIndexes: 'oprcesDataObjInit',
  31.         indexes: [
  32.           {
  33.             key: { flowId: 1, 'activityConfiguration.activityNameEn': 1 },
  34.             name: 'flowId_1_activityConfiguration.activityNameEn_1',
  35.             ns: 'likingtest.oprcesDataObjInit'
  36.           },
  37.           {
  38.             key: {
  39.               'oprceInfo.oprceInstID': 1,
  40.               'activityInfo.activityInstID': 1,
  41.               'workitemInfo.workItemID': 1
  42.             },
  43.             name: 'oprceInfo.oprceInstID_1_activityInfo.activityInstID_1_workitemInfo.workItemID_1',
  44.             ns: 'likingtest.oprcesDataObjInit'
  45.           }
  46.         ],
  47. ......第二天再看，还没创建完索引：
  48.       currentOpTime: '2023-09-15T09:16:16.460+08:00',
  49.       effectiveUsers: [ { user: 'admin', db: 'admin' } ],
  50.       runBy: [ { user: '__system', db: 'local' } ],
  51.       threaded: true,
  52.       opid: 'shard1:11312917',
  53.       lsid: {
  54.         id: new UUID("e78379ff-9664-46b1-9e87-2bdd4abc5c5f"),
  55.         uid: Binary.createFromBase64("O0CMtIVItQN4IsEOsJdrPL8s7jv5xwh5a/A5Qfvs2A8=", 0)
  56.       },
  57.       secs_running: Long("53877"),
  58.       microsecs_running: Long("53877330742"),
  59.       op: 'command',
  60.       ns: 'likingtest.oprcesDataObjInit',
  61.       redacted: false,
  62.       command: {
  63.         createIndexes: 'oprcesDataObjInit',
  64. ......第二天满24h，还没创建完索引：
  65.       currentOpTime: '2023-09-15T18:55:16.877+08:00',
  66.       effectiveUsers: [ { user: 'admin', db: 'admin' } ],
  67.       runBy: [ { user: '__system', db: 'local' } ],
  68.       threaded: true,
  69.       opid: 'shard1:11312917',
  70.       lsid: {
  71.         id: new UUID("e78379ff-9664-46b1-9e87-2bdd4abc5c5f"),
  72.         uid: Binary.createFromBase64("O0CMtIVItQN4IsEOsJdrPL8s7jv5xwh5a/A5Qfvs2A8=", 0)
  73.       },
  74.       secs_running: Long("88617"),
  75.       microsecs_running: Long("88617747875"),
  76.       op: 'command',
  77.       ns: 'likingtest.oprcesDataObjInit',
  78.       redacted: false,
  79.       command: {
  80.         createIndexes: 'oprcesDataObjInit',
  81.         indexes: [
  82.           {
  83.             key: { flowId: 1, 'activityConfiguration.activityNameEn': 1 },
  84.             name: 'flowId_1_activityConfiguration.activityNameEn_1',
  85.             ns: 'likingtest.oprcesDataObjInit'
  86.           },

以上可见，mongorestore 导入数据库的数据效率目前是基本可控、可接受的，至少对于1.2T的大集合是可以接受的，但是最后的索引创建实在过于缓慢，且没有找到合适的解决办法：索引需多并发执行创建，且确保索引生效，本次索引创建最后并未生效

■ 2023-09-15T19:02 第4次10并发导入测试，不恢复索引

  1. mongorestore --port=20000 -uadmin -p'passwd' --authenticationDatabase=admin --numInsertionWorkersPerCollection=10 --bypassDocumentValidation --nsInclude="likingtest.*" --nsFrom="likingtest.*" --nsTo="likingtest.*" --noIndexRestore /u01/nfs/xxxxx_mongodb/10.1.1.1/20230914 > 20230914.10.2.2.2-4.log 2>&1 &
  2. tail -100f /u01/nfs/xxxxx_mongodb/10.1.1.1/20230914.10.2.2.2-4.log
  3. 2023-09-15T19:02:59.747+0800    preparing collections to restore from
  4. ...
  5. 2023-09-15T21:24:36.145+0800    [########################]  likingtest.oprceDataObj  1208GB/1208GB  (100.0%)
  6. 2023-09-15T21:24:36.161+0800    finished restoring likingtest.oprceDataObj (53413481 documents, 0 failures)
  7. 2023-09-15T21:24:36.165+0800    97367732 document(s) restored successfully. 0 document(s) failed to restore.

以上可见，耗时：2h22m

结论

1、restore 时需设置大数据量 collection 多并发导入：--numInsertionWorkersPerCollection=8
2、不恢复索引：--noIndexRestore
3、数据恢复后，后台创建索引：本站搜索"MongoDB 重建索引"

原文链接：https://www.cnblogs.com/likingzi/p/17750673.html

 友情链接：直通硅谷 　点职佳 　北美留学生论坛

MongoDB热门文章

  • MongoDB 数组查询（$elemMatch）、更新操作(占位符$)详解
  • MongoDB 聚合分组取第一条记录的案例及实现
  • 利用“海底捞算法”在MongoDB中优雅地存储一棵树
  • 瞧一瞧！这儿实现了MongoDB的增量备份与还原（含部署代码）
  • MongoDB 安装
  • MongoDB Python官方驱动 PyMongo 的简单封装
  • 记一次Mongodb数据库查询之包含所有指定元素的数组或者都在指定元素的数组中 - du-blog
  • Java将日期类型Date时间戳转换为MongoDB的时间类型数据
  • MongoDB动态建表方案（官方原生驱动）
  • MongoDB的模糊查询操作(类关系型数据库的 like 和 not like)

    • MongoDB推荐文章

  • SpringBoot集成Mongodb文档数据库
  • 用Python脚本迁移MongoDB数据到金仓-kingbase数据库 - 惟-koko
  • 使用Docker搭建MongoDB 5.0版本副本集集群
  • MongoDB 中的事务
  • 【最佳实践】高可用mongodb集群(1分片+3副本):规划及部署
  • 【最佳实践】MongoDB导出导入数据
  • MongoDB从入门到实战之.NET Core使用MongoDB开发ToDoList系统（8）-Ant Design Blazor前端框架搭建
  • 基于SqlSugar的开发框架循序渐进介绍（27）-- 基于MongoDB的数据库操作整合
  • .NET Core MongoDB数据仓储和工作单元模式实操
  • .NET Core MongoDB数据仓储和工作单元模式封装

    • 本站QQ群：前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

    W3xue 的所有内容仅供测试，对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
    关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接（目前9元/月）请联系QQ：27243702 沸活量
    皖ICP备17017327号-2　皖公网安备34020702000426号