|
|
上周更新系统，看到Gitlab-CE有新版本，直接就更新了。今天写好代码要推送的时候，发现服务端竟然挂了，提示502错误。
/ u/ f! ^( n! H6 @6 K登陆到服务器，运行 gitlab-ctl status 查看状态，组件都正常运行。唯一的意外是：ssh登录服务器时，会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因，就先解决这个问题。
# s) S9 `4 K' W运行 locale -a 命令，发现系统中确实没有 en_US.utf8 的本地化设置文件。于是编辑 /etc/locale.gen 文件，取消 “en_US.UTF-8 UTF-8” 这一行的注释，运行 locale-gen ，生成 en_US.utf8 相关文件。
( D/ Z& g: a" D: Q' @接着设置语言和本地化设置：. f N o/ m) {0 S; t. k
export LC_ALL=en_US.UTF-8$ x0 B) `6 }3 K" _+ C8 I i
export LANG=en_US.UTF-8断开ssh重新登陆，bash的警告提示消失了。运行 gitlab-ctl restart 重启Gitlab，代码能正常推送，网站能打开，似乎问题都解决了。* T9 ~4 n' Q: i8 H
但是好事多磨，过了没多久，发现网页都是500错误。虽然能通过git正常推送和拉去代码，但是网页用不了也挺烦人的。想到上个月为了节省服务器资源，把Premethus、Grafana等用不到的组件都关了，会不会是这个原因导致的？
9 ~. I# o) W8 i7 ~/ t& L. B1 a0 g于是编辑 /etc/gitlab/gitlab.rb 文件，把组件又启用。运行 gitlab-ctl reconfigure 重新配置，意外发现了，脚本无法运行成功，出现了如下提示：* g5 ^, a0 o) m! F( N- R2 x
Recipe: gitlab::database_migrations
T" ^5 N. N# ^* A6 v0 j. B * ruby_block[check remote PG version] action nothing (skipped due to action :nothing): v/ \7 w0 b! a- `# x
* rails_migration[gitlab-rails] action run4 F& U& J T2 G" k" B& N* n9 Y
* bash[migrate gitlab-rails database] action run
+ p/ y' P% _- Z7 a+ V ! z. C. D7 c5 G+ e" L- v
================================================================================: E; I Z+ r& d
Error executing action `run` on resource 'bash[migrate gitlab-rails database]': e6 H V* G2 r2 y+ ?
================================================================================
% |% b0 I/ M3 T2 Q
$ N; v$ Q( N4 D/ ^& V% A Mixlib::ShellOut::ShellCommandFailed4 L( c% y# Q" A
------------------------------------
& [4 t1 ~$ d! A+ \8 [ Command execution failed. STDOUT/STDERR suppressed for sensitive resource8 x3 w/ O$ Z U; V
; ]2 r; G& [$ W8 { Resource Declaration:
# j5 x$ p# n0 t x! f3 M# a ---------------------. V9 y' V: r Y$ ?. ^4 S
suppressed sensitive resource output
5 j8 S9 b- y" o4 E( _: i" T+ f $ I9 c% C4 O H4 i7 x- g/ E7 C6 a
Compiled Resource:
2 K& w! n0 q9 k: b F$ Q- E ------------------
; t* d7 w+ L/ V( U. J suppressed sensitive resource output
' a; Q9 e5 c0 T$ t& X$ A
) v0 F$ j. X, w System Info:' \2 H, v# h( R
------------
, Y1 M" p: _. i% Q+ E chef_version=15.17.4
0 I8 q0 p* m3 X Q- L. N platform=ubuntu6 M( R- `. Z, c5 n
platform_version=20.04
* |( e) x4 N: t; z5 K/ R& F ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]& f. [5 _. F, q9 x/ J
program_name=/opt/gitlab/embedded/bin/chef-client
1 v- x0 }' a3 a( f executable=/opt/gitlab/embedded/bin/chef-client
0 U" r: A0 q+ b 7 F! @9 o* K/ h( q, \
- }" `: J: ~6 ^2 X# E
================================================================================* W4 y8 [' F1 v
Error executing action `run` on resource 'rails_migration[gitlab-rails]'6 r% i) X$ L c9 _8 L
================================================================================
! R/ c8 i- O1 q. I2 J" j 7 N9 T$ }4 c1 S
Mixlib::ShellOut::ShellCommandFailed
) s" p6 m! s1 {8 N ------------------------------------
- D6 A' Z4 k" P5 u5 a" c bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
: n# m7 Y0 G0 i% ?) r6 r' k9 I' B O
3 K" }2 F+ d; c# i% E/ e ~. s8 r Resource Declaration:
9 G3 s, b B" t/ _: P$ n& H ---------------------
) L' \* q. E4 ?4 n* k8 ^ k5 h # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
8 P$ e' _- D8 \, ^ 4 o" P( A$ V/ x
51: rails_migration "gitlab-rails" do
8 m7 X2 v' A$ R7 k 52: rake_task 'gitlab:db:configure'
4 ?, ]: u8 |, C6 o7 Q' T 53: logfile_prefix 'gitlab-rails-db-migrate'
; M' H5 S9 ~% n0 \$ F* K- c: z 54: helper migration_helper( K- m& U3 O4 f; [
55:
7 }4 t9 ]& V) [( ]/ T4 s 56: environment env_variables
! T& K) s" p$ e% {4 B) C4 n1 O 57: dependent_services dependent_services
2 K ^) m3 p; I- t! o: W 58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately. [' A" v' e. O1 Z- A% l; @0 h5 X
59: notifies :run, "ruby_block[check remote PG version]", :immediately
* k, v8 r$ j8 w' O 60: 9 R+ R' {5 ^- ?/ R5 ~9 J& d' A
61: only_if { migration_helper.attributes_node['auto_migrate'] }
P9 B/ G5 N1 I0 ~+ o- d 62: end
4 U1 V4 ]) B, E& J# |
1 e) U; F' X' z8 E [, U Compiled Resource:& [! J! q3 v' [6 o: l
------------------& n3 [1 S- _/ @1 N
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file', ]- [4 s, f0 c1 |
9 E& @+ d/ N! m; C' B! V
rails_migration("gitlab-rails") do
7 A) C: X) S4 _9 I action [:run]
3 a% \7 r# [6 `5 @- @ default_guard_interpreter :default5 G, i7 q0 _; d3 T
declared_type :rails_migration' ~" W* `) q1 z" l8 b
cookbook_name "gitlab"' N- w5 \* j- s; M- P
recipe_name "database_migrations"& F6 K: }- @, w7 F7 E
rake_task "gitlab:db:configure"+ b: `6 b) s" y+ b/ c+ S
logfile_prefix "gitlab-rails-db-migrate"3 x- P- H7 ^" ]3 O% t9 ~
helper "*sensitive value suppressed*"
. d+ I3 J7 t( _! G( i0 @2 E0 [ environment "*sensitive value suppressed*"( s' l5 B4 Y6 b& c8 E1 G
dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
/ r$ r2 E* M! n" M only_if { #code block }+ @' m; [8 Q; a7 M0 G- J
end
+ Q0 g. W0 {3 o5 [, C: P/ n+ z( @" g $ }# h0 m1 h) ~
System Info:% Q$ n' z# v" d2 Q& F$ O( g
------------: Q3 m5 i$ R1 q: j& i
chef_version=15.17.4+ I2 \& ?8 b+ e5 b) }3 r
platform=ubuntu! v5 Q5 S8 p$ o1 ^/ z* G T& e4 e
platform_version=20.04/ B5 c, m8 p1 i
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
" Z& c# v4 ^, ]* r" ] program_name=/opt/gitlab/embedded/bin/chef-client' ^- }& s" r% g5 M. @7 t9 c' T
executable=/opt/gitlab/embedded/bin/chef-client
) t! i9 D6 j+ a8 b7 _! ?7 r & _5 j8 k+ G2 Y. F. s! F
Running handlers:& a" d' i c% _4 I% ?3 \# s
There was an error running gitlab-ctl reconfigure:
) u. }% C) T( {9 r3 q8 W4 ^% E; s% xrails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
2 ]3 D/ Y1 G' u' C4 ~8 K* w2 H! tRunning handlers complete
/ \0 [, q& x$ C+ ~6 HChef Infra Client failed. 0 resources updated in 21 seconds网上一查，发现和上次碰到的 Gitlab更新设置总是500错误 问题一样，又是Gitlab官方的锅。但是这次解决问题比之前更麻烦，步骤为：
7 [5 e- p. u6 D1 s I1. 运行 gitlab-rake db:migrate，出现错误没关系，中间会输出一行用于终止后台任务的命令； z% O# b! \# [! \& v$ h; X+ A- K) l
2. 复制这条命令或者上一步控制台输出的指令： gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']。运行这条指令可能会失败，如果失败，请重新运行上面的指令，成功说明问题解决了；* R5 q' B* I7 w D( B1 t# d
3. 再次运行 gitlab-rake db:migrate，确认不会再报错；
$ r6 v$ p$ s7 t) a/ ^4. 运行 gitlab-ctl reconfigure;1 |/ u+ B' t, J2 L
5. 重启Gitlab：gitlab-ctl restart。
( z% h$ T$ O) _- }需要注意的是，这个bug是在Gitlab-CE 14.10.0中引入，在14.10.1版本被修复。如果运行过reconfigure但没解决错误，是无法直接升级到14.10.1版本的。: |1 Q6 q# r9 o* @* ~% l
参考1. Gitlab-ctl reconfigure doesn’t work after gitlab omnibus updated
3 @/ u2 z D, s2 o+ R% G2. gitlab-ctl reconfigure failing due to migration issue
* a: J9 m/ A2 z: [" R9 y' AAD：【加速器推荐】 搬瓦工官方代理服务Just My Socks，高速CN2 GIA线路流畅访问外网，被墙自动更换IP打赏赞(2) |
|
发表于 2022-9-11 20:40:14
