|
|
上周更新系统，看到Gitlab-CE有新版本，直接就更新了。今天写好代码要推送的时候，发现服务端竟然挂了，提示502错误。* \! Y, {$ i$ f. w0 ?
登陆到服务器，运行 gitlab-ctl status 查看状态，组件都正常运行。唯一的意外是：ssh登录服务器时，会出现”LANG cannot change locale (en_US.utf8)” 的提示。暂时想不到其他原因，就先解决这个问题。
) B) m+ w3 |: W! D# [$ b: p4 i# d运行 locale -a 命令，发现系统中确实没有 en_US.utf8 的本地化设置文件。于是编辑 /etc/locale.gen 文件，取消 “en_US.UTF-8 UTF-8” 这一行的注释，运行 locale-gen ，生成 en_US.utf8 相关文件。
. a |; n" D: k8 t) g接着设置语言和本地化设置：! @* R1 R) E. h
export LC_ALL=en_US.UTF-8
6 N; s( }2 e5 n$ Q( sexport LANG=en_US.UTF-8断开ssh重新登陆，bash的警告提示消失了。运行 gitlab-ctl restart 重启Gitlab，代码能正常推送，网站能打开，似乎问题都解决了。
- u$ G& U6 Q7 g/ O9 x但是好事多磨，过了没多久，发现网页都是500错误。虽然能通过git正常推送和拉去代码，但是网页用不了也挺烦人的。想到上个月为了节省服务器资源，把Premethus、Grafana等用不到的组件都关了，会不会是这个原因导致的？
8 U/ S0 \9 a* C1 M o$ T# [于是编辑 /etc/gitlab/gitlab.rb 文件，把组件又启用。运行 gitlab-ctl reconfigure 重新配置，意外发现了，脚本无法运行成功，出现了如下提示：. d1 g6 t9 q9 O: [7 S
Recipe: gitlab::database_migrations2 s; G- j& ]+ D% J( u
* ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
, G. ?% |# ]. ` * rails_migration[gitlab-rails] action run
1 a0 V& \% V6 l1 _* K * bash[migrate gitlab-rails database] action run
6 ^$ q- V6 P- i0 E& K& c$ N
) m p5 |, O0 w ================================================================================
' T# `3 K9 ]! z" } Error executing action `run` on resource 'bash[migrate gitlab-rails database]'$ |2 m, l0 d! Q7 m( h% b5 s
================================================================================- B; m& ~! e/ o) f, b7 m
! T0 Z$ \9 ^9 r+ x% g {- ]7 m
Mixlib::ShellOut::ShellCommandFailed
$ J- m2 A8 _) j3 F0 O7 a ------------------------------------6 M% j+ i* K. ?% j
Command execution failed. STDOUT/STDERR suppressed for sensitive resource: ?4 I' g4 }6 c% e& M9 K1 L6 Y
2 [$ w+ ^1 K9 k- E- P0 g0 E3 b* H Resource Declaration:3 L: r8 {3 |; t3 _3 [8 P* x5 A
---------------------
+ h: x: m n* r( c* G5 r1 ~4 m suppressed sensitive resource output- A0 s2 p6 X) K! y& r( ~, [3 n; \. r
# j, F5 U0 b, \: e4 d
Compiled Resource:; ?9 V" z/ `$ v7 g) C: P5 x
------------------
5 |9 M6 f+ l! q- z7 P suppressed sensitive resource output
d% {: b9 ?2 N
9 C e# G2 I. w e2 c System Info:/ Q' p2 m, j" N% t2 ?5 t8 a4 ^
------------
% J" Y* e" H" R3 M- ? V chef_version=15.17.4
$ ]$ J1 a) \; Z# q9 @' i platform=ubuntu$ u1 W5 W) y; e2 W c
platform_version=20.04
" `+ G5 y, @% D4 f$ @ ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]$ }8 A9 T& X/ _" ^3 W% g
program_name=/opt/gitlab/embedded/bin/chef-client
; {& V( Z# R# N" S3 d6 B executable=/opt/gitlab/embedded/bin/chef-client- z$ T0 A0 n6 P4 }7 }
0 B, m8 p9 H; T% l0 Y- `( K3 a x) r
+ Q% v0 [& h$ `6 Y7 o, ]
================================================================================7 `* t. X b: O
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
) h" r' D+ A8 w) w( ~( S3 C: ?/ x' _" q ================================================================================9 F. ^. I1 p- N) R* _* h3 Z
" B8 D5 J+ d- T& a7 A) H+ \/ X Mixlib::ShellOut::ShellCommandFailed: K2 l! T3 x7 I/ h: p
------------------------------------% \$ K" ~+ F0 ~* b( L: d: x0 r8 B9 e- F [
bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
) y4 M7 {; s' z$ ]# \9 k6 H' P ' \' l2 q: @; r3 O3 W# ?; [
Resource Declaration:
! y- k+ \! l. | ---------------------( ~4 E. c! @7 z7 x3 { ~
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
8 Y( M# U' Z; l% B) Q & l5 X' F* I" h: H4 d5 p6 O
51: rails_migration "gitlab-rails" do
$ m# i% U i; F: r7 e 52: rake_task 'gitlab:db:configure'
* r7 j8 r* m/ Z$ r( y: w 53: logfile_prefix 'gitlab-rails-db-migrate'
4 X/ V9 x) b/ o 54: helper migration_helper) ?$ M0 l7 @ h. _! @* u* ?
55:
?5 _; I( s3 f4 o 56: environment env_variables* h9 `, p- \7 a0 d6 H; `5 E
57: dependent_services dependent_services
8 e; y% @# r0 A! `" u3 r 58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately+ `+ j& O, M6 K, G9 L2 n2 _9 j
59: notifies :run, "ruby_block[check remote PG version]", :immediately
( t8 B, O2 O. @. X1 \" ~ 60: # M% m; p$ A. |% T* }, o
61: only_if { migration_helper.attributes_node['auto_migrate'] }
% ?; d( w; ~ R" N# G0 y 62: end
. p0 ]2 u# Z; |, q0 g- I4 _ + Y1 I. G/ I/ F
Compiled Resource:& ~ y: S8 m) h) Q, A( p
------------------8 p/ o: _* E2 s+ J, D
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'8 l$ n8 o% t' v& N e8 `
' s) A1 ~! o% W, t9 A6 o; ^ rails_migration("gitlab-rails") do
0 y% j! c1 X: b4 g3 y' V( v3 i k action [:run]) |/ P: d7 ^; I3 z
default_guard_interpreter :default
' Z3 [$ P+ `9 Z& w& E& g. ~! w declared_type :rails_migration
3 O1 ]& S/ i! j% w cookbook_name "gitlab"
( L7 e: K( A5 p2 z$ |2 \& Q recipe_name "database_migrations"
6 l. f/ u6 a3 m rake_task "gitlab:db:configure": p1 ^: o6 T4 @) c+ Q. u9 [
logfile_prefix "gitlab-rails-db-migrate"7 Z" c4 ]" Y7 k0 I
helper "*sensitive value suppressed*"2 _' D3 e8 Q; P) w3 H9 m
environment "*sensitive value suppressed*"+ D- Z, ?" V: I5 y Z
dependent_services ["runit_service[puma]", "sidekiq_service[sidekiq]"]0 R* D- a+ j# H( ~% d* l7 }: m
only_if { #code block }( g# |. k7 A, z8 x2 l+ `( u0 _
end
E& f( Y( p* }; c- u
& L/ W# Z2 F* `/ L, d: q; p System Info:3 x' [0 l+ S% D) I
------------
1 C0 Y0 Q1 ]: y chef_version=15.17.4
# V$ o! f1 s* y5 M. ^ platform=ubuntu
3 F, m6 o- a( s5 Z, w% w platform_version=20.04: b6 Y* Q0 ]& I
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
2 s! U6 Q: @7 b3 L) n3 u program_name=/opt/gitlab/embedded/bin/chef-client# S8 U L E' ~8 j/ _
executable=/opt/gitlab/embedded/bin/chef-client
j9 t& Z P0 ~" }6 D9 q' ?
) Z; J& ]0 M/ _* N) X( a uRunning handlers:& T9 w" x( ^# {8 v. B$ {- n
There was an error running gitlab-ctl reconfigure:8 I5 C7 f: T( Y
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource& z0 n, ^3 q. o- @7 Y
Running handlers complete
+ ?4 a; [8 N8 _( lChef Infra Client failed. 0 resources updated in 21 seconds网上一查，发现和上次碰到的 Gitlab更新设置总是500错误 问题一样，又是Gitlab官方的锅。但是这次解决问题比之前更麻烦，步骤为：, @8 S+ d+ g i7 Y
1. 运行 gitlab-rake db:migrate，出现错误没关系，中间会输出一行用于终止后台任务的命令；7 z4 o0 y* B* u P. d
2. 复制这条命令或者上一步控制台输出的指令： gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']。运行这条指令可能会失败，如果失败，请重新运行上面的指令，成功说明问题解决了；
' V. r, q. i4 C s! g( F& W. Y3. 再次运行 gitlab-rake db:migrate，确认不会再报错；; d: S6 ]: M- t3 q7 {) G
4. 运行 gitlab-ctl reconfigure;
0 T4 I$ c" h) |1 ^* `5. 重启Gitlab：gitlab-ctl restart。4 X# F8 y# B& [3 o% I
需要注意的是，这个bug是在Gitlab-CE 14.10.0中引入，在14.10.1版本被修复。如果运行过reconfigure但没解决错误，是无法直接升级到14.10.1版本的。
0 B! C& ]1 D; S. Z4 h参考1. Gitlab-ctl reconfigure doesn’t work after gitlab omnibus updated
6 T2 [% [ q2 h1 T% w2. gitlab-ctl reconfigure failing due to migration issue
/ z) f4 S5 O. }8 u) K1 ]AD：【加速器推荐】 搬瓦工官方代理服务Just My Socks，高速CN2 GIA线路流畅访问外网，被墙自动更换IP打赏赞(2) |
|
发表于 2022-9-11 20:40:14
