|
|
上周更新系统，看到Gitlab-CE有新版本，直接就更新了。今天写好代码要推送的时候，发现服务端竟然挂了，提示502错误。* Y O6 f! A" v3 V' p1 e# X; @
登陆到服务器，运行 gitlab-ctl status 查看状态，组件都正常运行。唯一的意外是：ssh登录服务器时，会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因，就先解决这个问题。/ c, i5 \9 ~6 W
运行 locale -a 命令，发现系统中确实没有 en_US.utf8 的本地化设置文件。于是编辑 /etc/locale.gen 文件，取消 “en_US.UTF-8 UTF-8” 这一行的注释，运行 locale-gen ，生成 en_US.utf8 相关文件。
/ H6 d2 J3 E' A. | T" u/ I7 |' m8 W接着设置语言和本地化设置：
K- a# B9 I P8 b5 ^ p6 v$ ?" Sexport LC_ALL=en_US.UTF-8, n5 G$ a" ~; J: e; Z
export LANG=en_US.UTF-8断开ssh重新登陆，bash的警告提示消失了。运行 gitlab-ctl restart 重启Gitlab，代码能正常推送，网站能打开，似乎问题都解决了。
1 O5 r9 Q9 o1 F: [% ]: u" E, u& b但是好事多磨，过了没多久，发现网页都是500错误。虽然能通过git正常推送和拉去代码，但是网页用不了也挺烦人的。想到上个月为了节省服务器资源，把Premethus、Grafana等用不到的组件都关了，会不会是这个原因导致的？
L Q9 q2 Z: ~8 ?, |& v于是编辑 /etc/gitlab/gitlab.rb 文件，把组件又启用。运行 gitlab-ctl reconfigure 重新配置，意外发现了，脚本无法运行成功，出现了如下提示：
( n- B' Z$ H; {3 rRecipe: gitlab::database_migrations
8 d5 m. U# c* l0 @1 ~ * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
* I8 y+ P( |1 R8 [2 k * rails_migration[gitlab-rails] action run6 c6 B8 c4 e' v0 m* u
* bash[migrate gitlab-rails database] action run* m. S9 {4 O+ ~# v: }
/ x( d2 g& U+ r' C% S% |* y5 Y ================================================================================
3 Z5 I `2 i6 [" J: C+ m" [" ? Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
# V: _/ h# o! a# A1 M, S* h, q ================================================================================
2 u E0 F6 R- s7 E" p) u/ x: K 6 D W& H: ~. z4 ?; o, A
Mixlib::ShellOut::ShellCommandFailed
4 T: j7 w/ l5 i$ A ------------------------------------9 I& [3 z( y% \/ y
Command execution failed. STDOUT/STDERR suppressed for sensitive resource
7 x0 E6 ^6 d8 T7 T
H" Q2 B. F6 v) } Resource Declaration:
, T4 ]& E# L; C& Z D' K* b ---------------------* D! _& N7 P% X ^; u8 p6 o! F
suppressed sensitive resource output h/ y# g* G, Z! ?* ]1 C* Z- Z
3 B4 ^2 K) e9 B! \9 i
Compiled Resource:3 s2 p8 K* j. `# V+ M
------------------% j: D" v3 B# ~/ ~' U8 \) y7 a5 R
suppressed sensitive resource output
; G' `; E1 B7 D# ~+ g
0 U* k: e" A. O- O- A1 O+ Q System Info:, h: D! f$ l0 ^9 K
------------
: c, j% _: F4 u" L+ Y" X chef_version=15.17.4/ [. z- j( l- [3 `& g
platform=ubuntu
+ K8 S; S: U7 ^& W$ x. q platform_version=20.04, t0 y2 S/ M: i: D9 Z% ^; o
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
3 {0 P' |, i; k O8 M0 u$ E program_name=/opt/gitlab/embedded/bin/chef-client! j& E- }! a6 T8 r. r
executable=/opt/gitlab/embedded/bin/chef-client
& n! `" {! Q6 ?5 n. Q ; \, `5 R, `( L* \$ ?2 {* t6 r
+ L! M1 K& C1 J; w
================================================================================
2 ~4 `4 C: y# { Error executing action `run` on resource 'rails_migration[gitlab-rails]'* P6 k6 x% Q8 u
================================================================================9 L! U0 T' a/ i l8 o* X
" k* {& d; l8 T) Q* ] Mixlib::ShellOut::ShellCommandFailed
3 L! @( P. R2 ]" _. x+ X6 v- F ------------------------------------
% T D3 y8 y, `7 f% p6 C* ^ bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource8 W$ q7 ]+ {$ z, _6 x' i' B
2 s, l5 `- g) ?6 d7 {- ]
Resource Declaration:+ t4 P. a( w* [4 Z) k3 w3 r0 ^
--------------------- C( V& z$ l9 N0 u% u ` z
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
7 Z) Y4 Y* b0 g0 I / ~6 j; v( ^! K5 C" Z$ Z. C
51: rails_migration "gitlab-rails" do
/ D$ Z4 l. k e4 W& q 52: rake_task 'gitlab:db:configure'8 K% T7 u9 f& c/ n6 U
53: logfile_prefix 'gitlab-rails-db-migrate'
% g S8 [' _/ U2 F8 n6 v 54: helper migration_helper
2 J8 ^2 o* i/ s0 ~ K% S5 n- b 55:
1 m) L/ ~8 J4 J |5 n9 c% V8 z9 | 56: environment env_variables
, V0 u+ q) F! Y1 X 57: dependent_services dependent_services
: Q$ c, j( y8 x! w, S( z 58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately( [! r( Q3 n$ c2 q$ ~# @
59: notifies :run, "ruby_block[check remote PG version]", :immediately
( L! l' `3 @: d* H- B 60: # k( c- b U2 `
61: only_if { migration_helper.attributes_node['auto_migrate'] }( x# ], i8 G% [8 ?1 V/ @8 C
62: end
: Y& j: I& k, H6 Z( b
) x2 R3 F5 k' ~# F6 j2 J Compiled Resource:
' d3 H- r3 A5 c. S- w, N ------------------4 R/ T* z* x2 H* G9 E
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
8 L0 j/ D& O1 ` ' W5 t' k/ \. m. w! ]. v) y
rails_migration("gitlab-rails") do
, |8 }1 V0 \; u action [:run]2 E2 Y: D- o! h% G
default_guard_interpreter :default4 l5 t# p; ?0 T9 P8 ~1 F" [' I$ P
declared_type :rails_migration" v0 O2 U# R" g$ A+ h
cookbook_name "gitlab"
* Z* [# [9 M- o recipe_name "database_migrations"
5 [ D& ]9 ^) J rake_task "gitlab:db:configure"7 _9 p" l" B3 w: ? `
logfile_prefix "gitlab-rails-db-migrate"
& j# k) L d5 Y9 |0 u helper "*sensitive value suppressed*"
8 d; t' b4 i' R environment "*sensitive value suppressed*"
7 V1 [0 _# u2 t4 c, W* |4 }0 X( y dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
7 P* K) p6 f; a only_if { #code block }
8 s( q7 x" Z W end# ~. S7 _' |% U# a, U$ o/ `
, W8 Q3 M; Y1 \. i
System Info:
* c8 F! b) x6 i$ Y+ O3 q ------------
* _3 s: E" o& M2 I) U# T* Q chef_version=15.17.49 u# @5 W& ?/ I# F, ^, D$ V# k
platform=ubuntu* W0 e" I/ b1 t0 n) x
platform_version=20.04. i. Z; C7 W( J+ g G# X
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]( ]* j B! A* m& U8 l1 }
program_name=/opt/gitlab/embedded/bin/chef-client
: @+ i- g5 m/ y2 ?+ V) L. k; j executable=/opt/gitlab/embedded/bin/chef-client
1 Q! _; W1 V7 Q/ d
7 L( Y0 ]8 i" E: j$ \Running handlers:$ ?1 n Y3 Q! a9 l: W
There was an error running gitlab-ctl reconfigure:
, D& x) {1 P" P( trails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
8 X1 e( ~- O9 ~9 ~Running handlers complete
1 u) N- P4 |" U, Z" DChef Infra Client failed. 0 resources updated in 21 seconds网上一查，发现和上次碰到的 Gitlab更新设置总是500错误 问题一样，又是Gitlab官方的锅。但是这次解决问题比之前更麻烦，步骤为：" m7 l" t( L4 L s
1. 运行 gitlab-rake db:migrate，出现错误没关系，中间会输出一行用于终止后台任务的命令；
9 q D: h+ X1 d2. 复制这条命令或者上一步控制台输出的指令： gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']。运行这条指令可能会失败，如果失败，请重新运行上面的指令，成功说明问题解决了；4 }- F- p( A) e+ k T
3. 再次运行 gitlab-rake db:migrate，确认不会再报错；
O! ?0 _8 g& e4. 运行 gitlab-ctl reconfigure;4 I! ?+ P4 i' N1 `0 P3 u- T
5. 重启Gitlab：gitlab-ctl restart。 @+ o( z5 Z5 `$ e
需要注意的是，这个bug是在Gitlab-CE 14.10.0中引入，在14.10.1版本被修复。如果运行过reconfigure但没解决错误，是无法直接升级到14.10.1版本的。
; |5 ]' }- t& V$ B7 ]' o参考1. Gitlab-ctl reconfigure doesn’t work after gitlab omnibus updated
% Q- V$ | E$ B# P2. gitlab-ctl reconfigure failing due to migration issue
/ B6 a2 \4 [4 M+ ~3 S& l( F" KAD：【加速器推荐】 搬瓦工官方代理服务Just My Socks，高速CN2 GIA线路流畅访问外网，被墙自动更换IP打赏赞(2) |
|
发表于 2022-9-11 20:40:14
