|
|
上周更新系统，看到Gitlab-CE有新版本，直接就更新了。今天写好代码要推送的时候，发现服务端竟然挂了，提示502错误。
% o( q' `3 R/ n" N, s/ U. R0 s登陆到服务器，运行 gitlab-ctl status 查看状态，组件都正常运行。唯一的意外是：ssh登录服务器时，会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因，就先解决这个问题。; X* e3 w, t1 s7 W$ x) k4 j
运行 locale -a 命令，发现系统中确实没有 en_US.utf8 的本地化设置文件。于是编辑 /etc/locale.gen 文件，取消 “en_US.UTF-8 UTF-8” 这一行的注释，运行 locale-gen ，生成 en_US.utf8 相关文件。
' O* j8 P" s' I& H7 N接着设置语言和本地化设置： S: Y; K( z6 H4 }4 P
export LC_ALL=en_US.UTF-8
/ S$ ^4 v' E l; ~export LANG=en_US.UTF-8断开ssh重新登陆，bash的警告提示消失了。运行 gitlab-ctl restart 重启Gitlab，代码能正常推送，网站能打开，似乎问题都解决了。% D- o9 J" u4 }$ T
但是好事多磨，过了没多久，发现网页都是500错误。虽然能通过git正常推送和拉去代码，但是网页用不了也挺烦人的。想到上个月为了节省服务器资源，把Premethus、Grafana等用不到的组件都关了，会不会是这个原因导致的？% m1 E6 A: ]: w' K: P
于是编辑 /etc/gitlab/gitlab.rb 文件，把组件又启用。运行 gitlab-ctl reconfigure 重新配置，意外发现了，脚本无法运行成功，出现了如下提示：
) U0 j: [1 s* ?, O H! FRecipe: gitlab::database_migrations, @2 t' m1 j0 F- v# J$ B) C0 B2 \
* ruby_block[check remote PG version] action nothing (skipped due to action :nothing)+ \" L: A- P+ z; c0 t6 l
* rails_migration[gitlab-rails] action run# m7 }" y! B" ~# Q* v* M) D9 Y
* bash[migrate gitlab-rails database] action run6 i* X! r- L1 x: W9 j! W: e- c: n
) {# d3 {+ D5 W
================================================================================
% o) h1 i$ R8 G& p% I4 S( g Error executing action `run` on resource 'bash[migrate gitlab-rails database]'" {; |& _" a2 M% A
================================================================================
0 a- v4 F0 K1 M6 \: E7 b6 T
3 g+ a% M; m. Y4 v8 G$ J Mixlib::ShellOut::ShellCommandFailed
/ W& S- \1 ^+ g3 c- a" f ------------------------------------
+ z: P; n% S H1 [3 O7 n, A Command execution failed. STDOUT/STDERR suppressed for sensitive resource
' n- Z' d0 Z c6 P6 ~/ V0 n
/ _6 V9 Z, z2 _- M) B/ ~" Y! Q Resource Declaration:# R; C @. Q( i
---------------------/ f- W" n8 c" s5 r/ c
suppressed sensitive resource output
7 B6 z/ |( R- i) Y " P7 A, E6 Q# M' b, f
Compiled Resource:% U$ P2 |+ W; x) P2 p' |
------------------$ n7 n$ G3 \$ [: d3 `) {
suppressed sensitive resource output
7 f x9 S& U, a- A9 I : U. z# A) g+ i9 R+ i- F7 w" Q
System Info:- r; M% ?; y" k) p H* s+ z
------------
8 A' j" f5 j i% O7 c( u8 Q. V: X chef_version=15.17.4
1 z9 v. c, \! r platform=ubuntu
) A/ Z& G- C& n" S# D% B platform_version=20.04' u6 p3 E2 y( v" I. ?6 w$ g
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]8 p+ B6 o4 p1 |4 O8 I4 U
program_name=/opt/gitlab/embedded/bin/chef-client
" j( H5 @5 X& l' g0 K executable=/opt/gitlab/embedded/bin/chef-client
" G$ P7 U2 M1 [4 _ ( F8 h, a& u- d6 Q* H- E) r
$ n. v7 M2 V5 [/ |
================================================================================
1 f- [7 g4 k0 N! z* b Error executing action `run` on resource 'rails_migration[gitlab-rails]'2 N' y2 ~" G8 f: \9 D8 I8 U
================================================================================ b7 `* M9 ^. Z$ b- W3 D
3 y0 c/ ]9 K- Y5 |% {7 \' u
Mixlib::ShellOut::ShellCommandFailed
4 ~ s- a" {! L$ b, z ------------------------------------
. @ y% v: y* I* J$ h bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource3 B$ r8 y, H) Q8 V1 D5 A
1 w5 ?9 k, Z% M3 U r3 D6 S! ^! _7 s
Resource Declaration:: k f4 B/ |# B2 l
---------------------
: s3 Z$ }: N: v# B) D d # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
3 z, @# `( s+ D* J' B
5 D- e) o0 N0 M! L 51: rails_migration "gitlab-rails" do& D7 l! ]' @* ]5 T( Q I1 t, |
52: rake_task 'gitlab:db:configure'
/ y- S, B# Q: _# { 53: logfile_prefix 'gitlab-rails-db-migrate') g9 G( G5 a1 t
54: helper migration_helper
2 j" G4 @& e, E: k8 } 55: ) T8 {4 b% z7 e1 H, v! w: o4 E& Y
56: environment env_variables
4 S0 N, Q3 B8 i8 V 57: dependent_services dependent_services
1 x! m S8 @5 S2 i: G6 _. \ 58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately& X+ J" u% d! b
59: notifies :run, "ruby_block[check remote PG version]", :immediately
' R+ |, v' R% n* Y 60: + [! M; K X3 v6 A* I
61: only_if { migration_helper.attributes_node['auto_migrate'] } x/ D$ o; Z( T8 w& N2 v
62: end
5 ]0 K, P/ G% k/ Q ' |2 ?# N0 e6 N- A! _. ]
Compiled Resource:. \. a3 v! R. v* ?% n6 T7 D
------------------2 i0 v0 |3 C3 [; P: ]5 I# k
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'/ P0 Q# X& Y) `3 A! I0 n* ?) B: M3 @# H M
' J; y) f' n2 ~3 u" G
rails_migration("gitlab-rails") do- M% {$ `3 y+ I9 O% K, v
action [:run] W6 Y2 d# Y% \. E" M: g" D
default_guard_interpreter :default! @! _$ k# L5 X
declared_type :rails_migration
6 [0 F5 ~8 X/ j cookbook_name "gitlab"
; e+ y: _. ^+ x+ t' e recipe_name "database_migrations"
$ H1 E7 E+ _+ |" B# Q. [ O" t rake_task "gitlab:db:configure"
7 t$ _) I+ y0 M% _ logfile_prefix "gitlab-rails-db-migrate"; k" E% z$ j8 P F
helper "*sensitive value suppressed*"! z, Z5 S1 D* J/ D H* }
environment "*sensitive value suppressed*"4 \6 D/ R1 Y7 n; J& c
dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]0 u C6 N! F( j
only_if { #code block }
. d2 ^7 w Z# s" W6 F5 n* i8 c end, V) h7 H# a- ~- r! M" } F
4 `, j3 ~6 H% w& I2 b3 a7 ^9 C" t System Info:
: k$ i9 c9 @) V6 t ------------6 E/ i3 j! G R6 [4 P8 _- M }
chef_version=15.17.4
2 ^) B) H* r; I( q0 j0 k2 Q platform=ubuntu
' o7 O; h! X! k1 P9 ? platform_version=20.04$ ^3 ?+ L6 t# `9 v! @9 @/ t6 T
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]8 }/ n8 E5 L' ?/ T/ b
program_name=/opt/gitlab/embedded/bin/chef-client* g. m r& c* I5 {. g% @+ t" \0 Q
executable=/opt/gitlab/embedded/bin/chef-client. C) p% S5 r, O. D' G- O9 P
2 Q6 L8 f6 k9 K4 URunning handlers:
, [ z5 e6 w9 pThere was an error running gitlab-ctl reconfigure:' r% o5 X' {7 @% O! F
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource; v* l6 {7 c! A7 o. S
Running handlers complete
, _7 v, B3 r! X3 k; @" Q. AChef Infra Client failed. 0 resources updated in 21 seconds网上一查，发现和上次碰到的 Gitlab更新设置总是500错误 问题一样，又是Gitlab官方的锅。但是这次解决问题比之前更麻烦，步骤为：3 A- s5 t1 l2 K) L- E
1. 运行 gitlab-rake db:migrate，出现错误没关系，中间会输出一行用于终止后台任务的命令；6 s! n- E* v# X! t# ?% S1 J
2. 复制这条命令或者上一步控制台输出的指令： gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']。运行这条指令可能会失败，如果失败，请重新运行上面的指令，成功说明问题解决了；. J0 |- g8 T8 }* y
3. 再次运行 gitlab-rake db:migrate，确认不会再报错；8 V% [$ v7 L! b/ @1 c+ A; c
4. 运行 gitlab-ctl reconfigure;- c. B2 Q3 c6 A- d9 F0 Z
5. 重启Gitlab：gitlab-ctl restart。; f2 T( N; P: |) O
需要注意的是，这个bug是在Gitlab-CE 14.10.0中引入，在14.10.1版本被修复。如果运行过reconfigure但没解决错误，是无法直接升级到14.10.1版本的。4 o/ }9 n8 S+ x$ {8 \$ m, E% }
参考1. Gitlab-ctl reconfigure doesn’t work after gitlab omnibus updated6 p" n7 F G( l& e5 w
2. gitlab-ctl reconfigure failing due to migration issue- y. [( s+ H$ i. `, z
AD：【加速器推荐】 搬瓦工官方代理服务Just My Socks，高速CN2 GIA线路流畅访问外网，被墙自动更换IP打赏赞(2) |
|
发表于 2022-9-11 20:40:14
