|
|
上周更新系统，看到Gitlab-CE有新版本，直接就更新了。今天写好代码要推送的时候，发现服务端竟然挂了，提示502错误。8 |/ ~% K4 G- P$ t( Q
登陆到服务器，运行 gitlab-ctl status 查看状态，组件都正常运行。唯一的意外是：ssh登录服务器时，会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因，就先解决这个问题。
! Y* F# g! n j; v" G$ n( E运行 locale -a 命令，发现系统中确实没有 en_US.utf8 的本地化设置文件。于是编辑 /etc/locale.gen 文件，取消 “en_US.UTF-8 UTF-8” 这一行的注释，运行 locale-gen ，生成 en_US.utf8 相关文件。5 W! k) i* F5 k. @
接着设置语言和本地化设置：
( L1 @6 h4 y+ I) M8 W1 Sexport LC_ALL=en_US.UTF-8
. c* S2 a) D8 g1 D0 t) Aexport LANG=en_US.UTF-8断开ssh重新登陆，bash的警告提示消失了。运行 gitlab-ctl restart 重启Gitlab，代码能正常推送，网站能打开，似乎问题都解决了。
* W0 a4 x6 e1 e; s) P但是好事多磨，过了没多久，发现网页都是500错误。虽然能通过git正常推送和拉去代码，但是网页用不了也挺烦人的。想到上个月为了节省服务器资源，把Premethus、Grafana等用不到的组件都关了，会不会是这个原因导致的？
G2 r$ U/ m* F( J/ K1 O/ s于是编辑 /etc/gitlab/gitlab.rb 文件，把组件又启用。运行 gitlab-ctl reconfigure 重新配置，意外发现了，脚本无法运行成功，出现了如下提示：: `3 q& |' O" a( ?( A; T0 a
Recipe: gitlab::database_migrations
" I* b' n" d4 T9 h2 C4 y * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
* q# r. Z7 t' Q * rails_migration[gitlab-rails] action run) `! `5 r0 t. ^% E3 Y' e" |8 d
* bash[migrate gitlab-rails database] action run: }9 N) ~+ Q% q- @3 ^
. R; t: w. a h. h6 R/ S1 n) v ================================================================================
& x( a& e& J$ x: m Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
: F: {( u. v4 Z ================================================================================
* F/ T0 r& _$ ~: E
+ q F# |% }8 F Mixlib::ShellOut::ShellCommandFailed
5 j0 p5 _! y" W0 ]$ s ------------------------------------
* ]5 F5 R/ J' P0 \- Q8 ` Command execution failed. STDOUT/STDERR suppressed for sensitive resource" l- O$ L0 _- U1 b+ k7 W
* R6 m* A8 m2 Y
Resource Declaration:" y& B3 z9 |0 c3 {, {/ c
---------------------
2 B- E) u+ y% {6 g suppressed sensitive resource output% B# a4 q8 l& o6 O" C) R
& ^3 }: R5 S1 L* O( e: A! v
Compiled Resource:8 L* P' i& f+ C8 v) S6 f
------------------
4 N) m2 F; A3 O- x5 O$ j suppressed sensitive resource output, C7 q* { \+ |4 Z
5 X, X! Q' A$ {1 e7 p
System Info:8 z9 ~. Q/ L/ q: z" _9 x
------------
1 V! \$ g- m( {6 u chef_version=15.17.49 @) i p. M5 z+ M
platform=ubuntu
# w% X# {. u( ?6 E- r, Z) r! i0 n' A platform_version=20.04+ V* C# o' i: _# l2 B
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
# I/ K, p, k6 a5 j0 Q, m3 t program_name=/opt/gitlab/embedded/bin/chef-client
! h! ?4 V O! `( L. k executable=/opt/gitlab/embedded/bin/chef-client7 t( O" n7 p; S6 X# {
# l/ S4 D1 X# M; K) q 2 t% q7 k5 Y4 J7 n# ^
================================================================================, z% m) o! `( e
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
u7 c+ |1 a/ Y G5 a ================================================================================0 ^2 p, }% ?6 J
8 U! C0 j3 \, A9 \+ o
Mixlib::ShellOut::ShellCommandFailed
# A8 [& g$ X2 `8 G! K1 M ------------------------------------4 y; M s) T* ~( z2 o
bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
( {# Q( I! e4 W# t- y
9 \- S9 `% ?; h2 L Resource Declaration:; `9 a, n l; b, W$ \
---------------------
6 W& }( \) K- i g0 V # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb- D8 ?& T4 f/ c; A+ A. D; H4 y# ~
, I5 j. }* ~4 g* J0 w7 T5 y5 m 51: rails_migration "gitlab-rails" do
3 G" |( B- y/ ^- t: N- h/ ~ 52: rake_task 'gitlab:db:configure'6 p5 I7 d! W; r" C
53: logfile_prefix 'gitlab-rails-db-migrate'' b) c( t s4 _
54: helper migration_helper
' }+ t7 O& Q4 G7 w9 `' D& w0 B 55:
, k8 f* C8 c! q 56: environment env_variables
/ u5 \' {8 b8 L, [1 C: y ] 57: dependent_services dependent_services
" }, d9 w, n! E1 a$ Q" z 58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately" R8 n& F* F/ j& T% s- F
59: notifies :run, "ruby_block[check remote PG version]", :immediately
& I4 H8 P& k+ m- ]0 u 60:
' f& b/ G! Z) B H$ l( M 61: only_if { migration_helper.attributes_node['auto_migrate'] }
+ D7 x( N- c- w+ v; p; V! h 62: end
; S- a" s5 u' C# Z4 p ' f4 H+ [9 { v8 x7 w; r' }1 ~
Compiled Resource:
" r1 P& K5 u) ~) x1 |' f: v ------------------1 Y4 ]6 |4 u% `/ e$ Q* D
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
2 |3 j5 n; X6 ` 2 x/ w9 c- G4 s$ S( i3 e5 C
rails_migration("gitlab-rails") do
# Q! {( W" B! c7 o8 X, R action [:run]8 r, S( b- h) C, Q5 b
default_guard_interpreter :default! @, n1 i* F6 ?
declared_type :rails_migration
A4 B- ^7 L- E* ]5 J* N cookbook_name "gitlab"
C: K2 r9 }4 D, Z recipe_name "database_migrations"
. E$ ^( Y+ i: _% ^: B6 S2 v rake_task "gitlab:db:configure"
5 n2 ]9 `/ n+ m( g0 b3 f4 r logfile_prefix "gitlab-rails-db-migrate"
" Q) @! U; p+ V helper "*sensitive value suppressed*"# E6 }& t" O" m! S7 a' n
environment "*sensitive value suppressed*"- ^' F: e {# s {+ F/ I
dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
6 @ Z: a. n' Z3 n only_if { #code block }, n7 D. E4 P* P: u; X- j
end+ D! ^- d6 E/ A5 ~/ }
4 x1 c- P3 t4 f2 J9 ]# R/ u+ O System Info:3 k2 n8 f, N& [1 h
------------
S, B1 W) C ?4 [/ j chef_version=15.17.4: [3 c. E1 }9 h( `
platform=ubuntu
$ s1 o: S; { s platform_version=20.04
/ i% R1 W: P3 O8 J! [! } ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
" ]) M0 |5 u" y9 l4 A, o program_name=/opt/gitlab/embedded/bin/chef-client
" A+ I2 A) \3 j5 ?( R. Z/ e" \2 o& I executable=/opt/gitlab/embedded/bin/chef-client
; c) [2 E. g8 n ?4 K " s1 O1 ], o# j+ d4 d
Running handlers:9 U* S3 P/ d) ?' p4 F
There was an error running gitlab-ctl reconfigure:, t8 \. g4 B# m) ?/ G
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
) s! ^; K" I4 ORunning handlers complete
0 c, j, E- r1 Y/ w- P# j! B% OChef Infra Client failed. 0 resources updated in 21 seconds网上一查，发现和上次碰到的 Gitlab更新设置总是500错误 问题一样，又是Gitlab官方的锅。但是这次解决问题比之前更麻烦，步骤为：
7 r9 A( }7 C) A$ D+ ]) r1. 运行 gitlab-rake db:migrate，出现错误没关系，中间会输出一行用于终止后台任务的命令；
/ }- V4 t# F0 f, S2. 复制这条命令或者上一步控制台输出的指令： gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']。运行这条指令可能会失败，如果失败，请重新运行上面的指令，成功说明问题解决了；
' A$ B% p7 a1 H% A: Z6 q) B3. 再次运行 gitlab-rake db:migrate，确认不会再报错；3 v( k' c% E. h6 ]0 m* m! l2 F
4. 运行 gitlab-ctl reconfigure;4 x# m: _1 l' B9 i8 _
5. 重启Gitlab：gitlab-ctl restart。* _4 m- c7 C, W) F0 J. |
需要注意的是，这个bug是在Gitlab-CE 14.10.0中引入，在14.10.1版本被修复。如果运行过reconfigure但没解决错误，是无法直接升级到14.10.1版本的。
' u- E: ~* S" \/ H/ X参考1. Gitlab-ctl reconfigure doesn’t work after gitlab omnibus updated! W1 O- a, T( s# K& u
2. gitlab-ctl reconfigure failing due to migration issue4 p+ x3 z; A* }, @5 b
AD：【加速器推荐】 搬瓦工官方代理服务Just My Socks，高速CN2 GIA线路流畅访问外网，被墙自动更换IP打赏赞(2) |
|
发表于 2022-9-11 20:40:14
