Docker build à partir de Dockerfile avec plus de mémoire
Comment docker build de Dockerfile avec plus de mémoire?
C'est une question différente de celle-ci Autoriser plus de mémoire lorsque Docker crée un Dockerfile
Lors de l'installation native du logiciel, il y a suffisamment de mémoire pour construire et installer avec succès marian tool
Mais lors de la construction de l'image Docker à l'aide de Dockerfilehttps://github.com/marian-nmt/marian/blob/master/scripts/docker/Dockerfile.cp , il échoue avec plusieurs erreurs de mémoire épuisée
virtual memory exhausted: Cannot allocate memory
[en dehors]:
Step : RUN cmake $MARIANPATH && make -j
---> Running in 4867d166d17a
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CUDA_TOOLKIT_ROOT_DIR not found or specified
-- Cannot find CUDA libraries. Compiling without them.
-- Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY)
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Boost version: 1.58.0
-- Found the following Boost libraries:
-- system
-- filesystem
-- program_options
-- timer
-- iostreams
-- python
-- thread
-- chrono
-- regex
-- date_time
-- atomic
-- Found Python
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found suitable version "2.7.12", minimum required is "2.7")
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.8")
-- Found Git: /usr/bin/git (found version "2.7.4")
-- Git version: 5abc774
-- Found SparseHash: /usr/include
-- Configuring done
-- Generating done
-- Build files have been written to: /marian/build
Scanning dependencies of target fast_align
Scanning dependencies of target extract_Lex
Scanning dependencies of target libcnpy
Scanning dependencies of target atools
Scanning dependencies of target libyaml-cpp-amun
[ 2%] Building CXX object src/amun/3rd_party/extract_Lex/CMakeFiles/extract_Lex.dir/exception.cpp.o
[ 2%] Building CXX object src/amun/3rd_party/extract_Lex/CMakeFiles/extract_Lex.dir/utils.cpp.o
[ 3%] Building CXX object src/amun/3rd_party/fast_align/CMakeFiles/fast_align.dir/src/ttables.cc.o
[ 6%] Building CXX object src/amun/3rd_party/fast_align/CMakeFiles/fast_align.dir/src/fast_align.cc.o
[ 7%] Building CXX object src/amun/3rd_party/extract_Lex/CMakeFiles/extract_Lex.dir/extract-Lex-main.cpp.o
[ 10%] Building CXX object src/amun/3rd_party/CMakeFiles/libcnpy.dir/cnpy/cnpy.cpp.o
[ 10%] Building CXX object src/amun/3rd_party/fast_align/CMakeFiles/atools.dir/src/atools.cc.o
[ 10%] Building CXX object src/amun/3rd_party/fast_align/CMakeFiles/atools.dir/src/alignment_io.cc.o
[ 11%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/regex_yaml.cpp.o
[ 12%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/scanner.cpp.o
[ 14%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/convert.cpp.o
[ 15%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/stream.cpp.o
[ 16%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/memory.cpp.o
[ 17%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/node.cpp.o
[ 19%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/directives.cpp.o
[ 20%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/null.cpp.o
[ 21%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emitfromevents.cpp.o
[ 23%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/singledocparser.cpp.o
[ 24%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emitterstate.cpp.o
[ 25%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/scantag.cpp.o
[ 26%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/simplekey.cpp.o
[ 28%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/binary.cpp.o
[ 29%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/parser.cpp.o
[ 30%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/nodeevents.cpp.o
[ 32%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emit.cpp.o
[ 33%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/exp.cpp.o
[ 35%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/nodebuilder.cpp.o
[ 38%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/scanscalar.cpp.o
[ 39%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emitter.cpp.o
[ 41%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/tag.cpp.o
[ 42%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/scantoken.cpp.o
[ 43%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/ostream_wrapper.cpp.o
Scanning dependencies of target libcommon
[ 37%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/node_data.cpp.o
[ 44%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/contrib/graphbuilder.cpp.o
[ 46%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/emitterutils.cpp.o
[ 34%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/contrib/graphbuilderadapter.cpp.o
[ 47%] Building CXX object src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/parse.cpp.o
[ 48%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/scorer.cpp.o
[ 50%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/git_version.cpp.o
[ 51%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/base_matrix.cpp.o
[ 52%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/sentence.cpp.o
[ 53%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/config.cpp.o
[ 55%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/utils.cpp.o
[ 56%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/vocab.cpp.o
[ 65%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/filter.cpp.o
[ 66%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/god.cpp.o
[ 67%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/exception.cpp.o
[ 69%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/search.cpp.o
[ 69%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/hypothesis.cpp.o
[ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/translation_task.cpp.o
[ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/printer.cpp.o
[ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/sentences.cpp.o
[ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/history.cpp.o
[ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/logging.cpp.o
[ 70%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/processor/bpe.cpp.o
[ 71%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/output_collector.cpp.o
[ 73%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/types.cpp.o
[ 74%] Building CXX object src/amun/CMakeFiles/libcommon.dir/common/loader.cpp.o
virtual memory exhausted: Cannot allocate memory
virtual memory exhausted: Cannot allocate memory
virtual memory exhausted: Cannot allocate memory
Et finalement, cela conduit à:
src/amun/CMakeFiles/libcommon.dir/build.make:254: recipe for target 'src/amun/CMakeFiles/libcommon.dir/common/loader.cpp.o' failed
virtual memory exhausted: Cannot allocate memory
make[2]: *** [src/amun/CMakeFiles/cpumode.dir/cpu/decoder/encoder_decoder.cpp.o] Error 1
src/amun/CMakeFiles/cpumode.dir/build.make:110: recipe for target 'src/amun/CMakeFiles/cpumode.dir/cpu/decoder/encoder_decoder.cpp.o' failed
[ 79%] Built target libcnpy
virtual memory exhausted: Cannot allocate memory
src/amun/CMakeFiles/libcommon.dir/build.make:326: recipe for target 'src/amun/CMakeFiles/libcommon.dir/common/printer.cpp.o' failed
make[2]: *** [src/amun/CMakeFiles/libcommon.dir/common/printer.cpp.o] Error 1
CMakeFiles/Makefile2:340: recipe for target 'src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/all' failed
make[1]: *** [src/amun/3rd_party/yaml-cpp/CMakeFiles/libyaml-cpp-amun.dir/all] Error 2
CMakeFiles/Makefile2:182: recipe for target 'src/amun/CMakeFiles/libcommon.dir/all' failed
make[1]: *** [src/amun/CMakeFiles/libcommon.dir/all] Error 2
make[1]: *** [src/amun/CMakeFiles/cpumode.dir/all] Error 2
CMakeFiles/Makefile2:110: recipe for target 'src/amun/CMakeFiles/cpumode.dir/all' failed
make: *** [all] Error 2
En regardant le docker build --help, il existe des options pour contrôler l'utilisation de la mémoire:
$ docker build --help
Usage: docker build [OPTIONS] PATH | URL | -
Build an image from a Dockerfile
Options:
--build-arg list Set build-time variables (default [])
--cache-from stringSlice Images to consider as cache sources
--cgroup-parent string Optional parent cgroup for the container
--compress Compress the build context using gzip
--cpu-period int Limit the CPU CFS (Completely Fair Scheduler) period
--cpu-quota int Limit the CPU CFS (Completely Fair Scheduler) quota
-c, --cpu-shares int CPU shares (relative weight)
--cpuset-cpus string CPUs in which to allow execution (0-3, 0,1)
--cpuset-mems string MEMs in which to allow execution (0-3, 0,1)
--disable-content-trust Skip image verification (default true)
-f, --file string Name of the Dockerfile (Default is 'PATH/Dockerfile')
--force-rm Always remove intermediate containers
--help Print usage
--isolation string Container isolation technology
--label list Set metadata for an image (default [])
-m, --memory string Memory limit
--memory-swap string Swap limit equal to memory plus swap: '-1' to enable unlimited swap
--network string Set the networking mode for the RUN instructions during build (default "default")
--no-cache Do not use cache when building the image
--pull Always attempt to pull a newer version of the image
-q, --quiet Suppress the build output and print image ID on success
--rm Remove intermediate containers after a successful build (default true)
--security-opt stringSlice Security options
--shm-size string Size of /dev/shm, default value is 64MB
-t, --tag list Name and optionally a tag in the 'name:tag' format (default [])
--ulimit ulimit Ulimit options (default [])
Mais je n'ai pas pu trouver la syntaxe correcte de l'endroit où placer exactement le -m option -_- |||
Ce n'est pas avant le Dockerfile:
# Before Docker file.
$ docker build -m 4g Dockerfile.cpu -t ibot-cpu .
"docker build" requires exactly 1 argument(s).
See 'docker build --help'.
Usage: docker build [OPTIONS] PATH | URL | -
Build an image from a Dockerfile
Ce n'est pas après Dockerfile avant -t
# Before -t
$ docker build Dockerfile.cpu -m 4g -t ibot-cpu .
"docker build" requires exactly 1 argument(s).
See 'docker build --help'.
Usage: docker build [OPTIONS] PATH | URL | -
Build an image from a Dockerfile
Ce n'est pas après -t avant le chemin local
# Before local path
$ docker build Dockerfile.cpu -t ibot-cpu -m 4g .
"docker build" requires exactly 1 argument(s).
See 'docker build --help'.
Usage: docker build [OPTIONS] PATH | URL | -
Build an image from a Dockerfile
Ce n'est pas après le chemin local à la fin aussi ...
# At the end...
$ docker build Dockerfile.cpu -t ibot-cpu . -m 4g
"docker build" requires exactly 1 argument(s).
See 'docker build --help'.
Usage: docker build [OPTIONS] PATH | URL | -
Build an image from a Dockerfile
Comment docker build de Dockerfile avec plus de mémoire?
Ma version docker:
docker version
Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Tue Mar 28 00:40:02 2017
OS/Arch: darwin/AMD64
Server:
Version: 17.04.0-ce
API version: 1.28 (minimum version 1.12)
Go version: go1.7.5
Git commit: 4845c56
Built: Wed Apr 5 18:45:47 2017
OS/Arch: linux/AMD64
Experimental: false
Ce n'est pas quelque chose d'ordre. Le Dockerfile doit être spécifié avec -f
docker build -f Dockerfile.cpu -t ibot-cpu -m 4g .
Comme vous pouvez le voir dans l'aide.
Cependant, sachez que par défaut docker ne limite pas la mémoire du conteneur. Cela peut prendre toute la mémoire libre.
Comme je peux voir que vous êtes sur OSX, qui exécute docker sur une machine virtuelle Linux. Configurez la mémoire maximale en cliquant sur l'icône de baleine dans la barre des tâches. Par défaut, c'est 2G.
Pour plus d'informations, veuillez consulter mon autre réponse: Comment attribuer plus de mémoire au conteneur Docker