|
|
上周更新系统，看到Gitlab-CE有新版本，直接就更新了。今天写好代码要推送的时候，发现服务端竟然挂了，提示502错误。
/ l8 {' K1 O q, S8 ]# \登陆到服务器，运行 gitlab-ctl status 查看状态，组件都正常运行。唯一的意外是：ssh登录服务器时，会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因，就先解决这个问题。
! n1 W* z3 B7 _9 w0 I! B/ y运行 locale -a 命令，发现系统中确实没有 en_US.utf8 的本地化设置文件。于是编辑 /etc/locale.gen 文件，取消 “en_US.UTF-8 UTF-8” 这一行的注释，运行 locale-gen ，生成 en_US.utf8 相关文件。5 B! u Z/ o3 I
接着设置语言和本地化设置：! O; K/ V4 @* l9 M3 B: ~
export LC_ALL=en_US.UTF-8
* n4 k( a9 F( `5 ^) I& ^+ Y. eexport LANG=en_US.UTF-8断开ssh重新登陆，bash的警告提示消失了。运行 gitlab-ctl restart 重启Gitlab，代码能正常推送，网站能打开，似乎问题都解决了。
' S: I/ f! ]8 i% f/ G0 N: h4 ~但是好事多磨，过了没多久，发现网页都是500错误。虽然能通过git正常推送和拉去代码，但是网页用不了也挺烦人的。想到上个月为了节省服务器资源，把Premethus、Grafana等用不到的组件都关了，会不会是这个原因导致的？. s4 M3 S. ?; W- F$ ?
于是编辑 /etc/gitlab/gitlab.rb 文件，把组件又启用。运行 gitlab-ctl reconfigure 重新配置，意外发现了，脚本无法运行成功，出现了如下提示：
* z& f; n7 o; `, [' L. ?Recipe: gitlab::database_migrations
$ L8 M5 U9 _; G( B, @ * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)6 q- J* x! |- Y' Y4 I; m& d" j
* rails_migration[gitlab-rails] action run
# h9 Y& T! ~$ w0 z * bash[migrate gitlab-rails database] action run! K- \( _- p+ P- T$ [/ T) o
; p" w# @! E6 q( J" m# b' D3 H- e ================================================================================
. x; z2 W! b& s* P Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
8 z v/ K4 k! }! p0 D, \2 j ================================================================================
' M! G! X5 ^2 L
* `5 ]6 K5 q) w( B8 h& Z Mixlib::ShellOut::ShellCommandFailed
2 S" v M& L. O0 p3 o ------------------------------------
6 L% r% B8 a6 N Command execution failed. STDOUT/STDERR suppressed for sensitive resource0 h% Z+ k% i( g# L
( p% i9 `+ o0 `9 n; J
Resource Declaration:
7 v- e1 R1 w/ J; O" } ---------------------- K5 m: ^( l0 E/ V0 [
suppressed sensitive resource output
7 r- p0 F5 ~' q. O+ X
0 x: Q t8 L7 K5 h, m2 T: @1 A4 N& h" ^ Compiled Resource:
' n$ Z7 w$ t" {& x" ^1 }; k& c# F ------------------
- `% }% V1 `3 F( Z- d/ u/ s9 t* R' U2 ] suppressed sensitive resource output
$ B9 u# e& d7 Z, E3 L
$ u* L" N# K% ~* T System Info: Z* N8 o$ e g
------------
" {3 [3 |( B6 \* f0 F8 h chef_version=15.17.4
9 t8 }4 r, v; ^8 S platform=ubuntu( v- b$ D1 {& I
platform_version=20.04- I& Z) U+ T+ A+ T
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]- l, G, d J; P
program_name=/opt/gitlab/embedded/bin/chef-client
1 v+ N' T9 x/ Z- U, @ executable=/opt/gitlab/embedded/bin/chef-client
: p! P! N, {) ~1 u5 e9 _4 M
# G* x0 ]% Q8 S U8 @# W; N ) }3 Y/ s% r7 t, X/ {7 }# k2 t
================================================================================
+ h% W% G$ g3 [4 V Error executing action `run` on resource 'rails_migration[gitlab-rails]'! d8 X7 W) H0 j, v/ F6 [* J
================================================================================
: i! t9 x8 g- g
. e. l. x4 G9 y% Q0 `! q+ k Mixlib::ShellOut::ShellCommandFailed: {) s' P1 ]; W
------------------------------------
1 y0 b7 I# y) u/ K) N) R% d5 ? bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
8 f8 m6 z2 Z+ s: J& Y
8 q1 Y I! s# m8 E P" i6 {4 P Resource Declaration: w( O6 X' g! E0 k; i+ u
---------------------$ |9 X( L, i) ]1 J1 y: }
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb. n" s1 f7 C9 N; L1 E9 {- O
. `2 h) j) }* V9 g4 |
51: rails_migration "gitlab-rails" do
$ h: J) y8 e: i- P& a* V& s 52: rake_task 'gitlab:db:configure'
) ?, I! B9 h* V3 I 53: logfile_prefix 'gitlab-rails-db-migrate'# K9 ]/ O4 T/ l7 \! z4 Y
54: helper migration_helper
$ Z' y' ^0 n0 F* H* J/ n ] 55: + o9 L2 @9 [/ h+ K" D0 j4 ^
56: environment env_variables* m" e) o6 J6 R5 s
57: dependent_services dependent_services
j2 X3 s0 V& L 58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
! K7 I0 b4 k" a1 ]0 z& r 59: notifies :run, "ruby_block[check remote PG version]", :immediately
6 N, v/ j, T7 X5 V- u) S3 ` 60:
$ [9 [& ^5 ^1 ]0 `1 D 61: only_if { migration_helper.attributes_node['auto_migrate'] }
# [1 E. v! ]6 e& t2 P! s! K 62: end$ Z# l# D$ |; i& z
# M6 I4 L4 }0 E" M4 Q, r Compiled Resource:
* K* z( l% W. q- v7 s, Y P% e8 ~- p ------------------
# y2 ?& }+ ]6 a1 Z, _" L# g- k # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file') A3 O# l/ Z! F5 O7 j
& |$ s/ h1 w; P2 Z* V rails_migration("gitlab-rails") do6 E8 i; H/ o" o4 |1 u, a
action [:run]
5 [. }- z4 y0 j& t5 n& P6 o% r. A! S default_guard_interpreter :default
; c6 I1 _! X/ `3 f6 L declared_type :rails_migration6 |$ K. D1 K& A) X) c
cookbook_name "gitlab"
' ~* h# E- B3 L( m. `& `# ] recipe_name "database_migrations": {- X4 E5 F* q5 P
rake_task "gitlab:db:configure"
; O) R& z1 G5 q- l1 K" l) g4 c2 R: c logfile_prefix "gitlab-rails-db-migrate"! Z) p$ J% f$ L0 y$ j8 g
helper "*sensitive value suppressed*"' U! a+ M. {0 {7 g
environment "*sensitive value suppressed*"* x5 }+ @, Q- Q
dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]
! S6 q& N8 l9 j+ n$ k0 ? only_if { #code block }8 H9 C6 t& h! i1 G+ {/ V; d1 b
end
% K* m" W( D b' W8 U' E m0 n6 D# H9 R; r$ h. ]! Y
System Info:
" k6 Z3 M% z% e2 K) V ------------
* f% N" G- k9 O+ A9 t chef_version=15.17.4$ `! o; y$ d* h
platform=ubuntu! `% Y8 A& p8 H2 z3 c
platform_version=20.04
+ r+ ~; t& t5 b5 l ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]6 K7 h& \7 ]* G# J# o$ K0 C
program_name=/opt/gitlab/embedded/bin/chef-client2 `$ i% n: T, n1 a3 E! b
executable=/opt/gitlab/embedded/bin/chef-client
1 Z, N1 i6 [) A7 I
' R* C) `7 B2 W4 T2 ORunning handlers:
# ]; {" z3 \5 ~* t+ `3 mThere was an error running gitlab-ctl reconfigure:
. y: f# p& w$ H- T5 E% v. rrails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource9 ]; q k, |* t! A* a, i5 n
Running handlers complete; u, ~" B' Z( y
Chef Infra Client failed. 0 resources updated in 21 seconds网上一查，发现和上次碰到的 Gitlab更新设置总是500错误 问题一样，又是Gitlab官方的锅。但是这次解决问题比之前更麻烦，步骤为：8 j; w0 \! o: E, ?2 _$ ?
1. 运行 gitlab-rake db:migrate，出现错误没关系，中间会输出一行用于终止后台任务的命令；# g. ^- O& f; k4 J" f8 z) x
2. 复制这条命令或者上一步控制台输出的指令： gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']。运行这条指令可能会失败，如果失败，请重新运行上面的指令，成功说明问题解决了；
% x2 Q4 T4 |% y! O- K; F6 Y+ ?3. 再次运行 gitlab-rake db:migrate，确认不会再报错；
7 L' s. s# k0 f+ |/ O& M+ b- r4. 运行 gitlab-ctl reconfigure;+ X( t! G( E/ S. h
5. 重启Gitlab：gitlab-ctl restart。
/ J: M9 }- R9 ]8 D0 J3 h1 F) f需要注意的是，这个bug是在Gitlab-CE 14.10.0中引入，在14.10.1版本被修复。如果运行过reconfigure但没解决错误，是无法直接升级到14.10.1版本的。
. {" L- F7 C6 }! Q, |9 p: L$ g参考1. Gitlab-ctl reconfigure doesn’t work after gitlab omnibus updated% E" p/ `; ]" a" R, q( O
2. gitlab-ctl reconfigure failing due to migration issue
6 H5 I2 l2 L3 @" nAD：【加速器推荐】 搬瓦工官方代理服务Just My Socks，高速CN2 GIA线路流畅访问外网，被墙自动更换IP打赏赞(2) |
|
发表于 2022-9-11 20:40:14
