The Complete Peerage records that he died by "breaking his neck". 1
  ----------------------------------
  After 18 Nov 1252
  James of Audley gives the king two marks of gold for having a charter of warren, market and a fair.
  Calendar of the Fine Rolls, 37 Hen. III, 84
  ----------------------------------
  26 Oct 1260, Westminster
  Grant to James le Alditheleg of £40 yearly at the Exchequer, until the king provide for him in lands of wards and escheats of equivalent 
    value.  
  Calendar of Patent Rolls, Hen. III, vol. 5, p. 98
  ----------------------------------
  16 Jun 1263, St. Paul's, London
  Promise to James de Aldithele, to whom the king committed the 
    castles of Shrewsbury and Bruges during pleasure, which by reason 
    of the disturbance of the realm, he has to munition, that the king 
    will allow him his reasonable costs in so doing on condition that he 
    answer for the same as well as for other expenses made whereof 
    he had like letters before at the Exchequer.
  25 Oct 1263, Windsor
  Grant to James de Aldithele of the wardship of the lands and heirs of Robert de Bello Campo, tenant in chief; saving to Alice late his 
    wife her dower....
  26 Oct 1263, Windsor
  Mandate to Alan la Zuche, justice on this side Trent, to let James 
    de Aldithele have in the forest of Brehull four oaks for timber to 
    repair his houses of Stratton.
  24 Dec 1263, Windsor
  Power to Roger de Mortuo Mari, James de Alditheley, and Hamo le 
    Estraunge, when the king will send to the ford of Muntgomery at 
    Hilary next to treat of peace and a truce with Llewelin son of Griffin ; 
    with power to the said James to swear on the king's soul that he will 
    observe it.
  Calendar of Patent Rolls, Hen. III, vol. 5, pp. 266, 292, 293, 305
  ----------------------------------
  13 May 1264, Lewes
  Grant to James de Aldithele of the wardship of the lands and heirs 
    of John son of Robert de Bello Campo, a minor, deceased, who 
    married Joan daughter of the said James, who is near her time or 
    has already given birth to an heir; with the marriage of the heirs, 
    and the marriage of the said Joan.
  Calendar of Patent Rolls, Hen. III, vol. 5, p. 317
  ----------------------------------
  23 May 1271, Westminster
  Protection with clause volumus until Michaelmas and for four years 
    after, for James de Aldithele and Ralph le Keu, going to Ireland on 
    business of Edward the king's son.
  Calendar of Patent Rolls, Hen. III, vol. 6, p. 537
  ----------------------------------
  29 Jul 1272, Westminster
  The king has taken the homage of James of Audley son and heir of James of Audley deceased for all the lands and tenements which the aforementioned James his father held in chief on the day he died, and has rendered those lands and tenements to him. Order to Master Richard de Clifford, escheator this side of the Trent, that having accepted security from the aforementioned James for rendering his reasonable relief at the Exchequer, to cause the same James to have full seisin without delay of all the aforesaid lands and tenements of which the aforementioned James his father was seised in his demesne as of fee in his bailiwick on the day he died and which, by reason of the death of the same James, were taken into the king’s hand.
  Calendar of the Fine Rolls, 56 Hen. III, 1156
  ----------------------------------
  Inquisition Post Mortem
  788. James de Aldithele alias de Audithele. 2
  Writ of certiorari, 16 July, 56 Hen. III [1272], on the complaint of Ella, late the wife of the said James, that the escheator had taken into the king's hand the manors of Stratton and Wechewik, which were of her free marriage of the gift of her father William Lungespei. Inq. (undated)
  James, his son, aged 22 and more, is his heir.
  OXFORD. Stretton and Wrechewic manors, are held of the fee of Henry de Laci, and were given in free marriage with the said Ella to the said James.
  C. Hen. III. File 41. (1.)