|
|
上周更新系统，看到Gitlab-CE有新版本，直接就更新了。今天写好代码要推送的时候，发现服务端竟然挂了，提示502错误。 ?6 ]# ~: }7 m- E
登陆到服务器，运行 gitlab-ctl status 查看状态，组件都正常运行。唯一的意外是：ssh登录服务器时，会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因，就先解决这个问题。
- j* K0 t" m( ^$ s运行 locale -a 命令，发现系统中确实没有 en_US.utf8 的本地化设置文件。于是编辑 /etc/locale.gen 文件，取消 “en_US.UTF-8 UTF-8” 这一行的注释，运行 locale-gen ，生成 en_US.utf8 相关文件。
- A- C- [8 E+ H接着设置语言和本地化设置：
% s4 |3 `6 h& @) l! U3 d/ _, Qexport LC_ALL=en_US.UTF-8
/ }6 E1 j, e1 ]3 ]9 q' zexport LANG=en_US.UTF-8断开ssh重新登陆，bash的警告提示消失了。运行 gitlab-ctl restart 重启Gitlab，代码能正常推送，网站能打开，似乎问题都解决了。
2 ]# Y& i8 K a7 t ~- _, x但是好事多磨，过了没多久，发现网页都是500错误。虽然能通过git正常推送和拉去代码，但是网页用不了也挺烦人的。想到上个月为了节省服务器资源，把Premethus、Grafana等用不到的组件都关了，会不会是这个原因导致的？
( t L9 D+ [3 L于是编辑 /etc/gitlab/gitlab.rb 文件，把组件又启用。运行 gitlab-ctl reconfigure 重新配置，意外发现了，脚本无法运行成功，出现了如下提示：
2 A9 v7 R, E6 M, hRecipe: gitlab::database_migrations
; N8 C7 G$ n, P; t. j) O9 M * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
0 O& |0 z0 W Y* l7 C * rails_migration[gitlab-rails] action run9 z! B! i2 g! k6 e9 Q, c% W0 h7 d
* bash[migrate gitlab-rails database] action run7 X$ M" e0 s9 W
# x! J+ d0 h+ H
================================================================================6 I" A' L/ F a5 R9 {- M
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'3 T. d4 m$ L& |* {) |- E
================================================================================
; [/ @& y5 N% R6 ^' k
; y: l+ f+ R2 Q Mixlib::ShellOut::ShellCommandFailed6 @! P9 ?- ]; o
------------------------------------
- V9 o$ l' `0 G3 V Command execution failed. STDOUT/STDERR suppressed for sensitive resource8 b: H. i* p8 n( j, ~ Z
, \" g, x8 y+ ]* z
Resource Declaration:
4 @ n$ K( C; D* f+ C6 i ---------------------) ~! O0 G6 `/ w# R, m2 T9 l
suppressed sensitive resource output
2 \% Q' O$ A4 i/ l( z# q4 O( p& C, r
. H7 u6 O K' ^3 l" M5 ^ Compiled Resource:0 i$ z& p' j) g, }% S
------------------
7 O6 d' ~$ `) P' I6 U# J suppressed sensitive resource output
3 Q4 j5 x7 |' X* o' J* a6 _1 R6 J- v
" C# s2 G+ i! y System Info:! b, U; ~9 ]7 ?' T U% U/ j
------------! s0 e# F) l; g' L
chef_version=15.17.4
+ o- T( A2 F8 J6 W& y; ~/ h platform=ubuntu
( g$ d& g9 C7 v( o! g3 A. c; R platform_version=20.047 w2 }' c( d& Y
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]. I" f5 t% s+ R* S) x
program_name=/opt/gitlab/embedded/bin/chef-client
2 v0 _- V+ K) f executable=/opt/gitlab/embedded/bin/chef-client, N7 s& s& q8 [* A1 H5 m0 }
% V, o( ^& \4 B: h
& r( Y, ^+ G# |% K; R: p ================================================================================! g) P) }/ u) m/ o: H' V3 N
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
( s& [. @+ y! g* T5 x ================================================================================
/ f/ r+ S( _ T# |. g( |5 C# k
9 j# Y8 u" o0 j7 n4 g" L Mixlib::ShellOut::ShellCommandFailed
$ `$ c3 Y! S6 z4 @/ { ------------------------------------
]( p# v$ {# @5 C, j9 h' |4 M- w bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource5 b- u% J$ _2 [) k
! F& F) R9 k* B4 Z' x) L
Resource Declaration:
# L6 R; U/ Z& ^6 { ---------------------3 _5 Q" E$ A8 x+ d0 ]
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb( W- e) z4 o$ T2 g7 Q7 U
2 }; r: q$ D8 A/ S8 ^ 51: rails_migration "gitlab-rails" do0 v0 y+ R# N1 B- }4 x; A. H7 i+ h! }
52: rake_task 'gitlab:db:configure'
, D' G/ n7 V0 e E7 s* j 53: logfile_prefix 'gitlab-rails-db-migrate'
8 @& z- j3 y5 x! W6 L) S7 O 54: helper migration_helper0 {, ]2 y- e4 ^* J% t9 o! a; I( p( C" Y
55: + m2 v5 s$ ?. }
56: environment env_variables
# X" k' q! x3 T8 l9 M3 w1 d3 j 57: dependent_services dependent_services- j1 G- T5 ^( e% ~( K
58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately% c+ E5 M$ J" H2 Q( U0 h* s
59: notifies :run, "ruby_block[check remote PG version]", :immediately' X! o4 d0 N/ g+ K
60: 5 c$ W' R7 f: T% z) {' S9 P k. Z
61: only_if { migration_helper.attributes_node['auto_migrate'] }
4 d. A0 P W/ U H 62: end
7 ]+ Y$ @6 n: o3 j
5 @1 [8 J" I3 ~5 P3 H Compiled Resource:" L. J) T+ v, I0 [+ V) k1 r
------------------7 J- u# _- q: A" j! `7 `' M3 ]
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
' h0 ~; b9 O3 J. I1 m: V' p4 t. v + U* q: r: z, d$ f$ q
rails_migration("gitlab-rails") do, `6 X7 x# Y+ D1 G: E
action [:run]
8 [0 \: i# L( ~9 b9 @3 e* x% Q- x, l default_guard_interpreter :default; v. ^" Y: @( `
declared_type :rails_migration& V x- r7 @8 d5 J0 H
cookbook_name "gitlab". @4 t( F: y7 } R, d5 h. M
recipe_name "database_migrations"( L- y, F+ k5 K9 Z# G2 O
rake_task "gitlab:db:configure"
( b+ E- t! o4 o logfile_prefix "gitlab-rails-db-migrate"
% Y9 v2 N" Y8 {8 ?0 W5 I helper "*sensitive value suppressed*"
% p; W) l& X$ F8 Z/ ]. E environment "*sensitive value suppressed*"
3 y5 R0 y( P- e! t! { dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
! O$ ?7 O) _. q8 e; A only_if { #code block }
2 o6 l6 ~% Q @ W2 Z' Q/ o) a end
2 Q% X- V4 v: S8 r) A& m) P
9 A6 ~3 S( b* c# o System Info:+ C0 a3 C' T! ^ O2 d: f" Z
------------! B' b @4 J2 |9 d* P
chef_version=15.17.4% y8 `& E s1 u# z1 H7 b/ a; @% m
platform=ubuntu% S+ ~8 R$ e& M0 }( l
platform_version=20.04
: h* e3 T( K' z9 m! E; f; a ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]) j# z. m8 @- I5 f
program_name=/opt/gitlab/embedded/bin/chef-client
1 P0 [' N$ A* N- Y$ I5 D executable=/opt/gitlab/embedded/bin/chef-client
9 Z+ S/ y3 }. D 0 }/ `. A/ R5 k! M" L. T$ S7 I
Running handlers:
6 E8 ^+ f* V( F1 |3 Z- Q% ZThere was an error running gitlab-ctl reconfigure:
. z- v4 L' p9 t' D9 y4 w8 crails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource' b4 O$ x" `, S, S. ~
Running handlers complete) y& ~8 `& q& e5 @, U: x
Chef Infra Client failed. 0 resources updated in 21 seconds网上一查，发现和上次碰到的 Gitlab更新设置总是500错误 问题一样，又是Gitlab官方的锅。但是这次解决问题比之前更麻烦，步骤为：, Y. f$ ~2 }2 h
1. 运行 gitlab-rake db:migrate，出现错误没关系，中间会输出一行用于终止后台任务的命令；: Y. M( n: W. J" e' S! k
2. 复制这条命令或者上一步控制台输出的指令： gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']。运行这条指令可能会失败，如果失败，请重新运行上面的指令，成功说明问题解决了；
: y9 Z6 ~. u" E; ?/ ~3. 再次运行 gitlab-rake db:migrate，确认不会再报错；
8 k8 @! Y/ c5 W4 v2 g Z4. 运行 gitlab-ctl reconfigure;' ~ U% ^5 r. o7 b- C
5. 重启Gitlab：gitlab-ctl restart。
0 n1 S x( a2 }' D1 T需要注意的是，这个bug是在Gitlab-CE 14.10.0中引入，在14.10.1版本被修复。如果运行过reconfigure但没解决错误，是无法直接升级到14.10.1版本的。& j9 e N l; G2 C5 n
参考1. Gitlab-ctl reconfigure doesn’t work after gitlab omnibus updated
, O, D/ X' M8 |: y7 @1 C2. gitlab-ctl reconfigure failing due to migration issue4 x( A( L, o% y# ?& G! }7 B
AD：【加速器推荐】 搬瓦工官方代理服务Just My Socks，高速CN2 GIA线路流畅访问外网，被墙自动更换IP打赏赞(2) |
|
发表于 2022-9-11 20:40:14
