I'd like to capture the data sent by my ADSL modem (what's really sent on the RJ11 cable).
That's what an oscilloscope is for.
... I'd like to build a mini-computer ...
A mini-computer is a low-cost computer (i.e. 16-bits) circa the 1970-80s. That term is now deprecated.
... with 2 RJ11 plug, one connected to the modem, one to the telephone plug, and be able to capture/edit the data sent and recieved.
Somebody knows where I should start ?
If you want to make any sense of the ADSL signal on the phone line, then you need a complementary ADSL transceiver. There are various standards for ADSL signals, but all involve rather complex phase and amplitude modulations to achieve multiple bits per symbol and a full duplex communications channel on just two wires. Full duplex means that the receiver on each end has an echo canceler for the that end's transmitter, and any "signal" that you "capture" off the wires will be a mix from the transmitters from both ends.
If you do manage to build such a unit that could be inserted into the phone line between the ISP's DSLAM and your ADSL modem, I'm sure there would be some telco equipment manufacturers that would be interested. Such a device could be used to extend the existing distance limitations of ADSL. I used to work on similar devices (a digital repeater) but for HDSL. The non-deployment of such ADSL devices should be a hint that the ADSL experts consider such a device very difficult to implement.
Why couldn't I use one to make the work ?
"One" what? Another modem?
You are asking about intercepting a comm link that uses a sophisticated modulation. The nice square-ish waves you see in textbooks to represent logic or "digital" signals are employing (simple) amplitude modulation (i.e. high and low voltage levels) to transmit the digital information (aka bits). You simply cannot extract or view (let alone "edit") the digital data without first demodulating the analog signal. ADSL uses one of the most complex modulation schemes (employing both phase and amplitude) among anything else that is connected to your computer.
Communication links like ADSL are point-to-point but asymmetrical, that is, the modem at your end (called the remote unit) is a "slave" unit, and the modem/DSLAM at the ISP or central office end (called the line unit) is a "master" unit. In order to use the ADSL link, the line and remote units first have to perform a predefined series transmissions and responses (called training) to evaluate line conditions and set transmission rates, power levels and equalization. Only after training is complete can the ADSL link be used to transmit/receive data. Bottom line is that you cannot use an ADSL modem only built for use as the remote unit in place of the ADSL line unit.