order by查询速度过慢

现在有ods_yallagame__t_account_full表,1.5亿条数据,主键模型(为了频繁的实时更新)建表语句如下:
CREATE TABLE ods_yallagame__t_account_full (
fid bigint(20) NOT NULL ,
fname varchar(65533) NULL ,
fnationalityid bigint(20) NULL ,
fheaderurl varchar(65533) NULL,
fstatus bigint(20) NULL,
flanguage bigint(20) NULL,
fage bigint(20) NULL,
fsex bigint(20) NULL,
fbirthday varchar(65533) NULL,
fexperience bigint(20) NULL,
fcurrency bigint(20) NULL,
fdiamond bigint(20) NULL,
flevelid bigint(20) NULL,
fskinid bigint(20) NULL,
fvipid bigint(20) NULL,
fregistersource bigint(20) NULL,
fregisterip varchar(65533) NULL,
fregistertime datetime NULL,
fdevice varchar(65533) NULL,
fdevicetype bigint(20) NULL,
fdownloadchannelid bigint(20),
fisdeleted tinyint(4) NULL,
fmodifytime datetime NULL,
fisused tinyint(4) NULL,
faccounttype bigint(20) NULL,
fvipvalue bigint(20) NULL,
fisrecharged tinyint(4) NULL,
fpushtoken varchar(65533) NULL,
fprettyid bigint(20) NULL,
fdeletestatus bigint(20) NULL,
fnationname varchar(65533) NULL
) ENGINE=OLAP
PRIMARY KEY(fid)
COMMENT “OLAP”
DISTRIBUTED BY HASH(fid) BUCKETS 12
PROPERTIES (
“replication_num” = “3”,
“in_memory” = “false”,
“storage_format” = “DEFAULT”
);

目前是3fe,3be,分开部署,starrcosk版本为StarRocks-2.0.1。be单节点查询内存8g上限,be机器皆为8核64g

现执行sql语句:SELECT fid,fprettyid,fname,fstatus,fcurrency,fdiamond,fregistertime FROM ods_ludo.ods_yallagame__t_account_full order by fid desc limit 0,30

需要耗费20s左右,据DBA反馈,在mysql也只需100ms即可出结果。

explain SELECT fid,fprettyid,fname,fstatus,fcurrency,fdiamond,fregistertime FROM ods_ludo.ods_yallagame__t_account_full order by fid desc limit 0,30;

±----------------------------------------------------------------------------------------------------------------+
| Explain String |
±----------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0 |
| OUTPUT EXPRS:1: fid | 29: fprettyid | 2: fname | 5: fstatus | 11: fcurrency | 12: fdiamond | 18: fregistertime |
| PARTITION: UNPARTITIONED |
| |
| RESULT SINK |
| |
| 2:MERGING-EXCHANGE |
| limit: 30 |
| use vectorized: true |
| |
| PLAN FRAGMENT 1 |
| OUTPUT EXPRS: |
| PARTITION: RANDOM |
| |
| STREAM DATA SINK |
| EXCHANGE ID: 02 |
| UNPARTITIONED |
| |
| 1:TOP-N |
| | order by: <slot 1> 1: fid DESC |
| | offset: 0 |
| | limit: 30 |
| | use vectorized: true |
| | |
| 0:OlapScanNode |
| TABLE: ods_yallagame__t_account_full |
| PREAGGREGATION: ON |
| partitions=1/1 |
| rollup: ods_yallagame__t_account_full |
| tabletRatio=12/12 |
| tabletList=112087,112091,112095,112099,112103,112107,112111,112115,112119,112123 … |
| cardinality=161284317 |
| avgRowSize=7.0 |
| numNodes=0 |
| use vectorized: true |
±----------------------------------------------------------------------------------------------------------------+

profile.txt (42.4 KB)

show tablet from tablename;

RowCount

13408868
13410402
13410059
13408462
13324011
13395999
13393112
13411763
13413020
13439473
13368837
13369010
13429821
13499006
13370538
13360618
13397063
13444342
13464985
13330424
13352652
13442660
13425735
13474350
13385379
13424256
13396903
13419144
13431400
13375248
13323134
13435317
13436618
13398128
13302819
13367066

因为是根据自增主键进行分桶的,所以数据分布还是挺均匀的

这个主要耗时在IO上。这个查询走的是扫描全表。

对的,这个好像没有啥办法了。重铸mysql荣光,吾辈义不容辞 :joy:

也不能说完全无解,搞一个sort 的 runtime filter 应该可以解决这个问题