English
« Back to projectatomic.io
Ask Your Question
0

Device-mapper error

asked 2015-02-03 09:34:26 +0000

pwFoo gravatar image

updated 2015-02-12 08:45:36 +0000

I can't pull images or start docker containers and found the following error message in the logs.

Feb 03 09:27:02 atomic01 kernel: device-mapper: space map metadata: unable to allocate new metadata block Feb 03 09:27:02 atomic01 kernel: device-mapper: space map common: dmtmshadowblock() failed Feb 03 09:27:02 atomic01 kernel: device-mapper: space map metadata: unable to allocate new metadata block Feb 03 09:27:02 atomic01 kernel: device-mapper: space map metadata: unable to allocate new metadata block Feb 03 09:27:02 atomic01 kernel: device-mapper: space map common: dmtmshadowblock() failed Feb 03 09:27:02 atomic01 kernel: device-mapper: thin: Creation of new snapshot 6510 of device 4 failed. Feb 03 09:27:02 atomic01 docker[918]: Error running DeviceCreate (createSnapDevice) dmtaskrun failed

Added information...

[root@atomic01 fedora]# docker info Containers: 2 Images: 29 Storage Driver: devicemapper Pool Name: docker-253:1-131-pool Pool Blocksize: 65.54 kB Data file: /dev/atomicos/docker-data Metadata file: /dev/atomicos/docker-meta Data Space Used: 4.035 GB Data Space Total: 4.123 GB Metadata Space Used: 8.262 MB Metadata Space Total: 8.389 MB Library Version: 1.02.93 (2015-01-30) Execution Driver: native-0.2 Kernel Version: 3.18.5-201.fc21.x86_64 Operating System: Fedora 21 (Twenty One) CPUs: 2 Total Memory: 3.86 GiB Name: atomic01 ID: 34TI:NSR3:GBZX:UKBX:EXPT:VCX4:SO6K:LUA7:B75B:FGZB:IHJQ:QULJ

Load an image

[root@atomic01 fedora]# docker pull centos:centos6 centos:centos6: The image you are pulling has been verified a30bc9f3097e: Download complete a30bc9f3097e: Error pulling image (centos6) from centos, Driver devicemapper failed to create image rootfs a30bc9f3097ea36bfd37fecf575c2c63f5ba99b618916422c13ee3f12ecbf2e8: Error running DeviceCreate (createSnapDevice) dmtaskrun failed reate (createSnapDevice) dmtaskrun failed 5b12ef8fd570: Download complete FATA[0062] Error pulling image (centos6) from centos, Driver devicemapper failed to create image rootfs a30bc9f3097ea36bfd37fecf575c2c63f5ba99b618916422c13ee3f12ecbf2e8: Error running DeviceCreate (createSnapDevice) dmtaskrun failed

After atomic upgrade a new error...

Feb 06 16:53:08 atomic01 kube-controller-manager[622]: I0206 16:53:08.545758 622 restclient.go:133] Waiting for completion of operation 1 Feb 06 16:53:10 atomic01 kube-controller-manager[622]: I0206 16:53:10.547307 622 restclient.go:133] Waiting for completion of operation 1 Feb 06 16:53:12 atomic01 kube-controller-manager[622]: I0206 16:53:12.548861 622 restclient.go:133] Waiting for completion of operation 1 Feb 06 16:53:14 atomic01 kube-controller-manager[622]: I0206 16:53:14.550060 622 restclient.go:133] Waiting for completion of operation 1 Feb 06 16:53:16 atomic01 kube-controller-manager[622]: I0206 16:53:16.551303 622 restclient.go:133] Waiting for completion of operation 1 Feb 06 16:53:18 atomic01 kube-controller-manager[622]: I0206 16:53:18.552617 622 restclient.go:133] Waiting for completion of operation 1 Feb 06 16:53:20 atomic01 kube-controller-manager[622]: I0206 16:53:20.553910 622 restclient.go:133] Waiting for completion of operation 1

Looks like Fedora Atomic (or kubernetes / docker?) have some bugs and stability problems?

edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
0

answered 2015-02-11 09:07:35 +0000

pwFoo gravatar image

updated 2015-02-11 17:05:19 +0000

After atomic upgrade there is a blocking operation

Feb 11 09:02:21 atomic01 kube-controller-manager[619]: I0211 09:02:21.275243     619 restclient.go:133] Waiting for completion of operation 1
Feb 11 09:02:23 atomic01 kube-controller-manager[619]: I0211 09:02:23.276409     619 restclient.go:133] Waiting for completion of operation 1
Feb 11 09:02:25 atomic01 kube-controller-manager[619]: I0211 09:02:25.278007     619 restclient.go:133] Waiting for completion of operation 1

docker info

Containers: 3
Images: 29
Storage Driver: devicemapper
 Pool Name: docker-253:1-131-pool
 Pool Blocksize: 65.54 kB
 Data file: /dev/atomicos/docker-data
 Metadata file: /dev/atomicos/docker-meta
 Data Space Used: 4.035 GB
 Data Space Total: 4.123 GB
 Metadata Space Used: 8.262 MB
 Metadata Space Total: 8.389 MB
 Library Version: 1.02.93 (2015-01-30)
Execution Driver: native-0.2
Kernel Version: 3.18.5-201.fc21.x86_64
Operating System: Fedora 21 (Twenty One)
CPUs: 2
Total Memory: 3.86 GiB
Name: atomic01.nh-webkreativ.de
ID: 34TI:NSR3:GBZX:UKBX:EXPT:VCX4:SO6K:LUA7:B75B:FGZB:IHJQ:QULJ

Storage is a mounted LVM.

/dev/mapper/storage-docker   20G  1.1G   19G   6% /var/lib/docker

So docker used the wrong partition...

 Data file: /dev/atomicos/docker-data
 Metadata file: /dev/atomicos/docker-meta
 Data Space Used: 4.035 GB
 Data Space Total: 4.123 GB

/dev/mapper/storage-docker   20G  1.1G   19G   6% /var/lib/docker

How to change it?! Used that howto: http://www.projectatomic.io/docs/docker-storage-recommendation/

The solution was found here!http://ask.projectatomic.io/en/question/170/docker-does-not-use-a-separate-storage-disk/

But! Solution only works if docker is started via command line with "docker -d". Started by systemd wrong size is detected... So I finally deleted /etc/sysconfig/docker-storage (moved it away...) and now it seems to work!!!

edit flag offensive delete link more

Comments

The solution in the other question specifies a clean storage device to be added via docker-storage-setup. If you have mounted storage at /var/lib/docker, the config won't see it. By removing /etc/sysconfig/docker-setup, you turned off LVM volumes that Atomic uses and set back to docker defaults.

nzwulfin ( 2015-02-24 15:05:30 +0000 )edit
0

answered 2015-02-05 16:25:51 +0000

That looks like the metadata device is full.

Can you post the output of docker info? That should tell you what the devices in the thin-pool for data and metadata are as well as space available/used.

Example:

[fedora@atomic-host-001 ~]$ sudo docker info
Containers: 0
Images: 42
Storage Driver: devicemapper
 Pool Name: docker-253:0-2999425-pool
 Pool Blocksize: 65.54 kB
 Data file: /dev/atomicos/docker-data
 Metadata file: /dev/atomicos/docker-meta
 Data Space Used: 1.877 GB
 Data Space Total: 4.123 GB
 Metadata Space Used: 2.101 MB
 Metadata Space Total: 8.389 MB
 Library Version: 1.02.93 (2015-01-30)
Execution Driver: native-0.2
Kernel Version: 3.18.3-201.fc21.x86_64
Operating System: Fedora 21 (Twenty One)
CPUs: 1
Total Memory: 489.7 MiB
Name: atomic-host-001.localdomain
ID: OLRP:DGWC:236Y:FVCH:TTK7:NTTS:HGHY:DUXY:O3DY:ODWY:DKD5:OAOO

How did you provide storage to the atomic host?

edit flag offensive delete link more

Comments

Added some information to initial post. Thanks for your reply.

pwFoo ( 2015-02-06 16:57:33 +0000 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

[hide preview]

Question Tools

Follow
1 follower

Stats

Asked: 2015-02-03 09:34:26 +0000

Seen: 1,920 times

Last updated: Feb 12 '15