Background
Recently, when running sparksql, I frequently print logs and report errors in the middle of execution:
WARN TaskMemoryManager: Failed to allocate a page (104876 bytes), try again.
WARN TaskMemoryManager: Failed to allocate a page (104876 bytes), try again.
…
reason
There is not enough memory to perform tasks, and resources need to be recycled frequently
Solution:
1. Optimize SQL scripts. (preferred, that’s how I solved it at that time)
2. Increase driver memory, — driver memory 6g
My SQL at that time was simplified as follows:
select name
from stu
where id in (select id from in_stu);
Stu data volume is 800W, in_stu data volume is 1.2kW
Optimized as:
select name
from stu
where id in (select distinct id from in_stu);
after optimization, The data volume of In_stu ID is reduced to 11 W, and the problem is solved.
